Homework3 - Stats 2620
Tyler Poelking, Cornell Blake
10/15/2017
Question 1
The lasso, relative to least squares: iii is true. Lasso is less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
Ridge regression relative to least squares: Again, iii is true. Ridge regression is less flexible and hence will give improved prediction accuracy when its increase in bias is less than its decrease in variance.
Non-linear methods relative to least squares: ii is true. Non- linear methods are more flexible and hence will give improved prediction accuracy when its increase in variance is less than its decrease in bias.
a-c)
#Part a
set.seed(42)
X = rnorm(n=100)
e = rnorm(n=100)
# B0=1 B1=3 B2=2 B3=1
B = c(1,3,2,1)
#Part B
Y = rep(B[1], 100)
Y = Y + (B[2]*X)+ (B[3]*X^2)+ (B[4]*X^3)+ e
#Part C
x_data = data.frame(X, X^2, X^3, X^4, X^5, X^6, X^7, X^8, X^9, X^10, Y)
colnames(x_data) =c('X', 'X^2', 'X^3', 'X^4', 'X^5', 'X^6', 'X^7', 'X^8', 'X^9', 'X^10', 'Y')
x_data## X X^2 X^3 X^4 X^5
## 1 1.37095845 1.879527064 2.576754e+00 3.532622e+00 4.843078e+00
## 2 -0.56469817 0.318884025 -1.800732e-01 1.016870e-01 -5.742247e-02
## 3 0.36312841 0.131862243 4.788293e-02 1.738765e-02 6.313950e-03
## 4 0.63286260 0.400515077 2.534710e-01 1.604123e-01 1.015190e-01
## 5 0.40426832 0.163432877 6.607074e-02 2.671031e-02 1.079813e-02
## 6 -0.10612452 0.011262413 -1.195218e-03 1.268419e-04 -1.346104e-05
## 7 1.51152200 2.284698749 3.453372e+00 5.219848e+00 7.889916e+00
## 8 -0.09465904 0.008960334 -8.481766e-04 8.028758e-05 -7.599945e-06
## 9 2.01842371 4.074034289 8.223127e+00 1.659776e+01 3.350130e+01
## 10 -0.06271410 0.003933058 -2.466582e-04 1.546895e-05 -9.701211e-07
## 11 1.30486965 1.702684815 2.221782e+00 2.899136e+00 3.782994e+00
## 12 2.28664539 5.228747152 1.195629e+01 2.733980e+01 6.251642e+01
## 13 -1.38886070 1.928934047 -2.679021e+00 3.720787e+00 -5.167654e+00
## 14 -0.27878877 0.077723177 -2.166835e-02 6.040892e-03 -1.684133e-03
## 15 -0.13332134 0.017774579 -2.369731e-03 3.159356e-04 -4.212096e-05
## 16 0.63595040 0.404432909 2.571993e-01 1.635660e-01 1.040198e-01
## 17 -0.28425292 0.080799723 -2.296756e-02 6.528595e-03 -1.855772e-03
## 18 -2.65645542 7.056755403 -1.874596e+01 4.979780e+01 -1.322856e+02
## 19 -2.44046693 5.955878829 -1.453513e+01 3.547249e+01 -8.656945e+01
## 20 1.32011335 1.742699246 2.300561e+00 3.037001e+00 4.009185e+00
## 21 -0.30663859 0.094027227 -2.883238e-02 8.841119e-03 -2.711028e-03
## 22 -1.78130843 3.173059737 -5.652198e+00 1.006831e+01 -1.793476e+01
## 23 -0.17191736 0.029555577 -5.081117e-03 8.735321e-04 -1.501753e-04
## 24 1.21467470 1.475434625 1.792173e+00 2.176907e+00 2.644234e+00
## 25 1.89519346 3.591758256 6.807077e+00 1.290073e+01 2.444937e+01
## 26 -0.43046913 0.185303673 -7.976751e-02 3.433745e-02 -1.478121e-02
## 27 -0.25726938 0.066187535 -1.702803e-02 4.380790e-03 -1.127043e-03
## 28 -1.76316309 3.108744065 -5.481223e+00 9.664290e+00 -1.703972e+01
## 29 0.46009735 0.211689576 9.739781e-02 4.481248e-02 2.061810e-02
## 30 -0.63999488 0.409593441 -2.621377e-01 1.677668e-01 -1.073699e-01
## 31 0.45545012 0.207434815 9.447621e-02 4.302920e-02 1.959766e-02
## 32 0.70483734 0.496795672 3.501601e-01 2.468059e-01 1.739580e-01
## 33 1.03510352 1.071439301 1.109051e+00 1.147982e+00 1.188280e+00
## 34 -0.60892638 0.370791331 -2.257846e-01 1.374862e-01 -8.371898e-02
## 35 0.50495512 0.254979677 1.287533e-01 6.501464e-02 3.282947e-02
## 36 -1.71700868 2.948118804 -5.061946e+00 8.691404e+00 -1.492322e+01
## 37 -0.78445901 0.615375936 -4.827372e-01 3.786875e-01 -2.970649e-01
## 38 -0.85090759 0.724043734 -6.160943e-01 5.242393e-01 -4.460792e-01
## 39 -2.41420765 5.828398577 -1.407096e+01 3.397023e+01 -8.201119e+01
## 40 0.03612261 0.001304843 4.713432e-05 1.702615e-06 6.150288e-08
## 41 0.20599860 0.042435423 8.741638e-03 1.800765e-03 3.709551e-04
## 42 -0.36105730 0.130362373 -4.706829e-02 1.699435e-02 -6.135933e-03
## 43 0.75816324 0.574811492 4.358009e-01 3.304083e-01 2.505034e-01
## 44 -0.72670483 0.528099906 -3.837728e-01 2.788895e-01 -2.026704e-01
## 45 -1.36828104 1.872193017 -2.561686e+00 3.505107e+00 -4.795971e+00
## 46 0.43281803 0.187331444 8.108043e-02 3.509307e-02 1.518891e-02
## 47 -0.81139318 0.658358886 -5.341879e-01 4.334364e-01 -3.516874e-01
## 48 1.44410126 2.085428454 3.011570e+00 4.349012e+00 6.280413e+00
## 49 -0.43144620 0.186145826 -8.031191e-02 3.465027e-02 -1.494973e-02
## 50 0.65564788 0.429874147 2.818461e-01 1.847918e-01 1.211583e-01
## 51 0.32192527 0.103635876 3.336301e-02 1.074039e-02 3.457604e-03
## 52 -0.78383894 0.614403485 -4.815934e-01 3.774916e-01 -2.958926e-01
## 53 1.57572752 2.482917217 3.912401e+00 6.164878e+00 9.714168e+00
## 54 0.64289931 0.413319517 2.657228e-01 1.708330e-01 1.098284e-01
## 55 0.08976065 0.008056974 7.231992e-04 6.491482e-05 5.826797e-06
## 56 0.27655075 0.076480316 2.115069e-02 5.849239e-03 1.617611e-03
## 57 0.67928882 0.461433296 3.134465e-01 2.129207e-01 1.446346e-01
## 58 0.08983289 0.008069948 7.249467e-04 6.512405e-05 5.850282e-06
## 59 -2.99309008 8.958588246 -2.681386e+01 8.025630e+01 -2.402143e+02
## 60 0.28488295 0.081158297 2.312062e-02 6.586669e-03 1.876430e-03
## 61 -0.36723464 0.134861283 -4.952574e-02 1.818757e-02 -6.679104e-03
## 62 0.18523056 0.034310362 6.355328e-03 1.177201e-03 2.180536e-04
## 63 0.58182373 0.338518850 1.969583e-01 1.145950e-01 6.667410e-02
## 64 1.39973683 1.959263186 2.742453e+00 3.838712e+00 5.373187e+00
## 65 -0.72729206 0.528953740 -3.847039e-01 2.797921e-01 -2.034905e-01
## 66 1.30254263 1.696617308 2.209916e+00 2.878510e+00 3.749382e+00
## 67 0.33584812 0.112793960 3.788164e-02 1.272248e-02 4.272820e-03
## 68 1.03850610 1.078494917 1.120024e+00 1.163151e+00 1.207940e+00
## 69 0.92072857 0.847741096 7.805394e-01 7.186650e-01 6.616954e-01
## 70 0.72087816 0.519665326 3.746154e-01 2.700521e-01 1.946746e-01
## 71 -1.04311894 1.088097120 -1.135015e+00 1.183955e+00 -1.235006e+00
## 72 -0.09018639 0.008133584 -7.335386e-04 6.615519e-05 -5.966298e-06
## 73 0.62351816 0.388774898 2.424082e-01 1.511459e-01 9.424223e-02
## 74 -0.95352336 0.909206794 -8.669499e-01 8.266570e-01 -7.882368e-01
## 75 -0.54282881 0.294663122 -1.599516e-01 8.682636e-02 -4.713185e-02
## 76 0.58099650 0.337556930 1.961194e-01 1.139447e-01 6.620146e-02
## 77 0.76817874 0.590098573 4.533012e-01 3.482163e-01 2.674924e-01
## 78 0.46376759 0.215080376 9.974731e-02 4.625957e-02 2.145369e-02
## 79 -0.88577630 0.784599649 -6.949798e-01 6.155966e-01 -5.452809e-01
## 80 -1.09978090 1.209518025 -1.330205e+00 1.462934e+00 -1.608907e+00
## 81 1.51270701 2.288282498 3.461501e+00 5.236237e+00 7.920892e+00
## 82 0.25792144 0.066523468 1.715783e-02 4.425372e-03 1.141398e-03
## 83 0.08844023 0.007821674 6.917507e-04 6.117859e-05 5.410648e-06
## 84 -0.12089654 0.014615973 -1.767021e-03 2.136267e-04 -2.582672e-05
## 85 -1.19432890 1.426421510 -1.703616e+00 2.034678e+00 -2.430075e+00
## 86 0.61199690 0.374540203 2.292174e-01 1.402804e-01 8.585115e-02
## 87 -0.21713985 0.047149713 -1.023808e-02 2.223095e-03 -4.827226e-04
## 88 -0.18275671 0.033400014 -6.104076e-03 1.115561e-03 -2.038762e-04
## 89 0.93334633 0.871135369 8.130710e-01 7.588768e-01 7.082949e-01
## 90 0.82177311 0.675311045 5.549525e-01 4.560450e-01 3.747655e-01
## 91 1.39211638 1.937988004 2.697905e+00 3.755798e+00 5.228507e+00
## 92 -0.47617392 0.226741605 -1.079684e-01 5.141176e-02 -2.448094e-02
## 93 0.65034856 0.422953250 2.750670e-01 1.788895e-01 1.163405e-01
## 94 1.39111046 1.935188302 2.692061e+00 3.744954e+00 5.209644e+00
## 95 -1.11078888 1.233851935 -1.370549e+00 1.522391e+00 -1.691055e+00
## 96 -0.86079259 0.740963878 -6.378162e-01 5.490275e-01 -4.725988e-01
## 97 -1.13173868 1.280832442 -1.449568e+00 1.640532e+00 -1.856653e+00
## 98 -1.45921400 2.129305496 -3.107112e+00 4.533942e+00 -6.615991e+00
## 99 0.07998255 0.006397209 5.116651e-04 4.092428e-05 3.273228e-06
## 100 0.65320434 0.426675909 2.787066e-01 1.820523e-01 1.189174e-01
## X^6 X^7 X^8 X^9 X^10
## 1 6.639659e+00 9.102696e+00 1.247942e+01 1.710876e+01 2.345540e+01
## 2 3.242637e-02 -1.831111e-02 1.034025e-02 -5.839120e-03 3.297341e-03
## 3 2.292775e-03 8.325716e-04 3.023304e-04 1.097848e-04 3.986597e-05
## 4 6.424756e-02 4.065988e-02 2.573211e-02 1.628489e-02 1.030610e-02
## 5 4.365342e-03 1.764770e-03 7.134404e-04 2.884214e-04 1.165996e-04
## 6 1.428546e-06 -1.516038e-07 1.608888e-08 -1.707424e-09 1.811996e-10
## 7 1.192578e+01 1.802608e+01 2.724682e+01 4.118416e+01 6.225077e+01
## 8 7.194035e-07 -6.809804e-08 6.446095e-09 -6.101812e-10 5.775916e-11
## 9 6.761982e+01 1.364855e+02 2.754855e+02 5.560464e+02 1.122337e+03
## 10 6.084027e-08 -3.815543e-09 2.392883e-10 -1.500675e-11 9.411349e-13
## 11 4.936314e+00 6.441247e+00 8.404987e+00 1.096741e+01 1.431104e+01
## 12 1.429529e+02 3.268826e+02 7.474645e+02 1.709186e+03 3.908303e+03
## 13 7.177152e+00 -9.968064e+00 1.384425e+01 -1.922774e+01 2.670465e+01
## 14 4.695173e-04 -1.308962e-04 3.649238e-05 -1.017367e-05 2.836304e-06
## 15 5.615623e-06 -7.486824e-07 9.981533e-08 -1.330751e-08 1.774176e-09
## 16 6.615146e-02 4.206905e-02 2.675383e-02 1.701411e-02 1.082013e-02
## 17 5.275087e-04 -1.499459e-04 4.262256e-05 -1.211559e-05 3.443891e-06
## 18 3.514109e+02 -9.335073e+02 2.479821e+03 -6.587533e+03 1.749949e+04
## 19 2.112699e+02 -5.155971e+02 1.258298e+03 -3.070834e+03 7.494269e+03
## 20 5.292579e+00 6.986804e+00 9.223373e+00 1.217590e+01 1.607357e+01
## 21 8.313060e-04 -2.549105e-04 7.816539e-05 -2.396853e-05 7.349675e-06
## 22 3.194734e+01 -5.690807e+01 1.013708e+02 -1.805727e+02 3.216557e+02
## 23 2.581775e-05 -4.438519e-06 7.630584e-07 -1.311830e-07 2.255263e-08
## 24 3.211884e+00 3.901395e+00 4.738926e+00 5.756253e+00 6.991975e+00
## 25 4.633629e+01 8.781624e+01 1.664288e+02 3.154147e+02 5.977719e+02
## 26 6.362856e-03 -2.739013e-03 1.179061e-03 -5.075492e-04 2.184843e-04
## 27 2.899537e-04 -7.459620e-05 1.919132e-05 -4.937339e-06 1.270226e-06
## 28 3.004380e+01 -5.297212e+01 9.339849e+01 -1.646768e+02 2.903520e+02
## 29 9.486334e-03 4.364637e-03 2.008158e-03 9.239482e-04 4.251061e-04
## 30 6.871618e-02 -4.397800e-02 2.814569e-02 -1.801310e-02 1.152829e-02
## 31 8.925755e-03 4.065236e-03 1.851512e-03 8.432715e-04 3.840681e-04
## 32 1.226121e-01 8.642160e-02 6.091317e-02 4.293388e-02 3.026140e-02
## 33 1.229993e+00 1.273170e+00 1.317863e+00 1.364125e+00 1.412010e+00
## 34 5.097870e-02 -3.104227e-02 1.890246e-02 -1.151021e-02 7.008868e-03
## 35 1.657741e-02 8.370848e-03 4.226903e-03 2.134396e-03 1.077774e-03
## 36 2.562329e+01 -4.399542e+01 7.554051e+01 -1.297037e+02 2.227024e+02
## 37 2.330352e-01 -1.828066e-01 1.434043e-01 -1.124948e-01 8.824753e-02
## 38 3.795722e-01 -3.229809e-01 2.748269e-01 -2.338523e-01 1.989867e-01
## 39 1.979920e+02 -4.779939e+02 1.153977e+03 -2.785939e+03 6.725835e+03
## 40 2.221644e-09 8.025158e-11 2.898896e-12 1.047157e-13 3.782604e-15
## 41 7.641623e-05 1.574164e-05 3.242755e-06 6.680030e-07 1.376077e-07
## 42 2.215424e-03 -7.998948e-04 2.888079e-04 -1.042762e-04 3.764968e-05
## 43 1.899225e-01 1.439922e-01 1.091696e-01 8.276839e-02 6.275195e-02
## 44 1.472815e-01 -1.070302e-01 7.777936e-02 -5.652264e-02 4.107527e-02
## 45 6.562236e+00 -8.978983e+00 1.228577e+01 -1.681039e+01 2.300134e+01
## 46 6.574035e-03 2.845361e-03 1.231524e-03 5.330256e-04 2.307031e-04
## 47 2.853567e-01 -2.315365e-01 1.878671e-01 -1.524341e-01 1.236840e-01
## 48 9.069553e+00 1.309735e+01 1.891390e+01 2.731359e+01 3.944359e+01
## 49 6.450003e-03 -2.782829e-03 1.200641e-03 -5.180120e-04 2.234943e-04
## 50 7.943721e-02 5.208284e-02 3.414800e-02 2.238907e-02 1.467934e-02
## 51 1.113090e-03 3.583319e-04 1.153561e-04 3.713604e-05 1.195503e-05
## 52 2.319322e-01 -1.817975e-01 1.424999e-01 -1.116970e-01 8.755246e-02
## 53 1.530688e+01 2.411947e+01 3.800572e+01 5.988666e+01 9.436506e+01
## 54 7.060862e-02 4.539423e-02 2.918392e-02 1.876232e-02 1.206228e-02
## 55 5.230170e-07 4.694635e-08 4.213934e-09 3.782455e-10 3.395156e-11
## 56 4.473516e-04 1.237154e-04 3.421359e-05 9.461795e-06 2.616666e-06
## 57 9.824869e-02 6.673924e-02 4.533522e-02 3.079571e-02 2.091918e-02
## 58 5.255477e-07 4.721147e-08 4.241142e-09 3.809941e-10 3.422580e-11
## 59 7.189832e+02 -2.151981e+03 6.441074e+03 -1.927872e+04 5.770293e+04
## 60 5.345629e-04 1.522878e-04 4.338421e-05 1.235942e-05 3.520989e-06
## 61 2.452798e-03 -9.007526e-04 3.307875e-04 -1.214766e-04 4.461043e-05
## 62 4.039019e-05 7.481498e-06 1.385802e-06 2.566929e-07 4.754737e-08
## 63 3.879257e-02 2.257044e-02 1.313202e-02 7.640519e-03 4.445435e-03
## 64 7.521048e+00 1.052749e+01 1.473571e+01 2.062612e+01 2.887114e+01
## 65 1.479971e-01 -1.076371e-01 7.828360e-02 -5.693504e-02 4.140840e-02
## 66 4.883730e+00 6.361267e+00 8.285821e+00 1.079264e+01 1.405787e+01
## 67 1.435019e-03 4.819483e-04 1.618614e-04 5.436086e-05 1.825699e-05
## 68 1.254453e+00 1.302757e+00 1.352921e+00 1.405017e+00 1.459118e+00
## 69 6.092418e-01 5.609464e-01 5.164793e-01 4.755373e-01 4.378408e-01
## 70 1.403367e-01 1.011657e-01 7.292811e-02 5.257228e-02 3.789821e-02
## 71 1.288258e+00 -1.343807e+00 1.401750e+00 -1.462192e+00 1.525240e+00
## 72 5.380788e-07 -4.852739e-08 4.376510e-09 -3.947016e-10 3.559671e-11
## 73 5.876174e-02 3.663901e-02 2.284509e-02 1.424433e-02 8.881597e-03
## 74 7.516022e-01 -7.166702e-01 6.833618e-01 -6.516014e-01 6.213172e-01
## 75 2.558452e-02 -1.388802e-02 7.538816e-03 -4.092287e-03 2.221411e-03
## 76 3.846282e-02 2.234676e-02 1.298339e-02 7.543304e-03 4.382633e-03
## 77 2.054820e-01 1.578469e-01 1.212546e-01 9.314521e-02 7.155217e-02
## 78 9.949525e-03 4.614267e-03 2.139948e-03 9.924384e-04 4.602607e-04
## 79 4.829969e-01 -4.278272e-01 3.789592e-01 -3.356731e-01 2.973312e-01
## 80 1.769445e+00 -1.946002e+00 2.140175e+00 -2.353724e+00 2.588581e+00
## 81 1.198199e+01 1.812524e+01 2.741818e+01 4.147567e+01 6.274053e+01
## 82 2.943911e-04 7.592977e-05 1.958392e-05 5.051112e-06 1.302790e-06
## 83 4.785190e-07 4.232033e-08 3.742819e-09 3.310158e-10 2.927511e-11
## 84 3.122361e-06 -3.774827e-07 4.563635e-08 -5.517277e-09 6.670197e-10
## 85 2.902309e+00 -3.466311e+00 4.139916e+00 -4.944421e+00 5.905265e+00
## 86 5.254064e-02 3.215471e-02 1.967858e-02 1.204323e-02 7.370420e-03
## 87 1.048183e-04 -2.276023e-05 4.942153e-06 -1.073138e-06 2.330211e-07
## 88 3.725975e-05 -6.809469e-06 1.244476e-06 -2.274364e-07 4.156552e-08
## 89 6.610844e-01 6.170207e-01 5.758940e-01 5.375086e-01 5.016817e-01
## 90 3.079722e-01 2.530833e-01 2.079770e-01 1.709099e-01 1.404492e-01
## 91 7.278691e+00 1.013278e+01 1.410601e+01 1.963721e+01 2.733729e+01
## 92 1.165718e-02 -5.550847e-03 2.643169e-03 -1.258608e-03 5.993163e-04
## 93 7.566188e-02 4.920659e-02 3.200144e-02 2.081209e-02 1.353511e-02
## 94 7.247191e+00 1.008164e+01 1.402468e+01 1.950988e+01 2.714039e+01
## 95 1.878405e+00 -2.086511e+00 2.317673e+00 -2.574446e+00 2.859665e+00
## 96 4.068095e-01 -3.501786e-01 3.014312e-01 -2.594697e-01 2.233496e-01
## 97 2.101246e+00 -2.378062e+00 2.691344e+00 -3.045899e+00 3.447161e+00
## 98 9.654147e+00 -1.408747e+01 2.055663e+01 -2.999652e+01 4.377134e+01
## 99 2.618012e-07 2.093953e-08 1.674797e-09 1.339545e-10 1.071402e-11
## 100 7.767734e-02 5.073918e-02 3.314305e-02 2.164919e-02 1.414134e-02
## Y
## 1 12.64964835
## 2 0.80835140
## 3 1.39778400
## 4 5.80157088
## 5 1.93896805
## 6 0.80846987
## 7 13.13508002
## 8 0.61074520
## 9 23.61466017
## 10 0.93863812
## 11 10.51666779
## 12 30.38179379
## 13 -2.47316994
## 14 -0.20680543
## 15 -1.02788366
## 16 3.59158255
## 17 -0.22677713
## 18 -8.89992060
## 19 -10.30688467
## 20 10.88355528
## 21 -1.25431877
## 22 -5.12043964
## 23 0.66298036
## 24 8.39042732
## 25 20.67435104
## 26 -0.42882644
## 27 -0.27013271
## 28 -5.57790175
## 29 1.67632108
## 30 -0.18341901
## 31 3.44331681
## 32 3.96538614
## 33 7.35730265
## 34 0.81190856
## 35 4.59343376
## 36 -4.41384777
## 37 -0.72268191
## 38 0.48076877
## 39 -9.12651981
## 40 1.05855516
## 41 1.62550099
## 42 -0.75719445
## 43 4.41522963
## 44 -0.53713230
## 45 -2.33601217
## 46 3.86758341
## 47 -1.13264251
## 48 12.08156152
## 49 0.69450371
## 50 3.05216961
## 51 2.16571208
## 52 -2.15584805
## 53 15.77258753
## 54 3.74741408
## 55 0.81827376
## 56 0.76551123
## 57 4.26641748
## 58 0.48608132
## 59 -17.40944773
## 60 3.32776132
## 61 -0.05703297
## 62 0.55888536
## 63 3.78267406
## 64 11.49745127
## 65 0.08134099
## 66 11.94320081
## 67 1.27832141
## 68 7.84718198
## 69 6.32310540
## 70 5.47214611
## 71 -1.31795543
## 72 1.58159354
## 73 2.14545663
## 74 0.78035252
## 75 0.66566615
## 76 3.46344676
## 77 3.48902741
## 78 3.56421953
## 79 -0.29991550
## 80 -1.21686709
## 81 13.72764289
## 82 1.33986011
## 83 1.65046252
## 84 0.95942965
## 85 -1.71301947
## 86 2.47805189
## 87 1.13339063
## 88 1.06662245
## 89 5.51907413
## 90 3.77630572
## 91 11.95518855
## 92 -0.42809498
## 93 4.32463092
## 94 10.44176619
## 95 -2.19438222
## 96 0.34750864
## 97 -0.87934387
## 98 -1.63965586
## 99 3.06848219
## 100 4.22049282a<-regsubsets(Y~., data = x_data, nvmax = 10)
mysummary(a)## Subset selection object
## Call: regsubsets.formula(Y ~ ., data = x_data, nvmax = 10)
## 10 Variables (and intercept)
## Forced in Forced out
## X FALSE FALSE
## `X^2` FALSE FALSE
## `X^3` FALSE FALSE
## `X^4` FALSE FALSE
## `X^5` FALSE FALSE
## `X^6` FALSE FALSE
## `X^7` FALSE FALSE
## `X^8` FALSE FALSE
## `X^9` FALSE FALSE
## `X^10` FALSE FALSE
## 1 subsets of each size up to 10
## Selection Algorithm: exhaustive
## X `X^2` `X^3` `X^4` `X^5` `X^6` `X^7` `X^8` `X^9` `X^10`
## 1 ( 1 ) "*" " " " " " " " " " " " " " " " " " "
## 2 ( 1 ) " " " " "*" "*" " " " " " " " " " " " "
## 3 ( 1 ) "*" "*" "*" " " " " " " " " " " " " " "
## 4 ( 1 ) "*" "*" "*" " " "*" " " " " " " " " " "
## 5 ( 1 ) "*" "*" " " " " "*" " " "*" " " "*" " "
## 6 ( 1 ) "*" "*" " " "*" "*" " " "*" " " "*" " "
## 7 ( 1 ) "*" "*" " " " " "*" "*" "*" "*" "*" " "
## 8 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*" " "
## 9 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*" "*"
## 10 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*"
## [1] "Cps: "
## [1] 1043.992234 433.630846 21.318293 19.561749 5.568909
## [6] 6.763693 5.528537 7.337124 9.080219 11.000000
## [1] "Optimal model variable count:"
## [1] 7
## [1] "BICs: "
## [1] -153.7239 -226.1567 -375.3712 -374.1372 -385.3111 -381.5702 -380.5148
## [8] -376.1237 -371.8065 -367.2914
## [1] "Optimal model variable count:"
## [1] 5
## [1] "R-squared: "
## [1] 0.8039415 0.9092568 0.9805113 0.9811573 0.9839078 0.9840463 0.9846027
## [8] 0.9846356 0.9846798 0.9846936
## [1] "Optimal model variable count:"
## [1] 10The 7th model is best according to Cp. The 5th model is best according to BIC. The 8th model is best according to R-Squared.
d)
a2<-regsubsets(Y~., data = x_data, method = "forward", nvmax = 10)
mysummary(a2)## Subset selection object
## Call: regsubsets.formula(Y ~ ., data = x_data, method = "forward",
## nvmax = 10)
## 10 Variables (and intercept)
## Forced in Forced out
## X FALSE FALSE
## `X^2` FALSE FALSE
## `X^3` FALSE FALSE
## `X^4` FALSE FALSE
## `X^5` FALSE FALSE
## `X^6` FALSE FALSE
## `X^7` FALSE FALSE
## `X^8` FALSE FALSE
## `X^9` FALSE FALSE
## `X^10` FALSE FALSE
## 1 subsets of each size up to 10
## Selection Algorithm: forward
## X `X^2` `X^3` `X^4` `X^5` `X^6` `X^7` `X^8` `X^9` `X^10`
## 1 ( 1 ) "*" " " " " " " " " " " " " " " " " " "
## 2 ( 1 ) "*" "*" " " " " " " " " " " " " " " " "
## 3 ( 1 ) "*" "*" "*" " " " " " " " " " " " " " "
## 4 ( 1 ) "*" "*" "*" " " "*" " " " " " " " " " "
## 5 ( 1 ) "*" "*" "*" " " "*" " " " " " " "*" " "
## 6 ( 1 ) "*" "*" "*" " " "*" " " "*" " " "*" " "
## 7 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " "*" " "
## 8 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " "*" "*"
## 9 ( 1 ) "*" "*" "*" "*" "*" "*" "*" " " "*" "*"
## 10 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*"
## [1] "Cps: "
## [1] 1043.992234 710.611218 21.318293 19.561749 13.822436
## [6] 7.469641 8.763127 7.984529 9.697202 11.000000
## [1] "Optimal model variable count:"
## [1] 6
## [1] "BICs: "
## [1] -153.7239 -183.9605 -375.3712 -374.1372 -376.8578 -380.8120 -376.9656
## [8] -375.4016 -371.1162 -367.2914
## [1] "Optimal model variable count:"
## [1] 6
## [1] "R-squared: "
## [1] 0.8039415 0.8616211 0.9805113 0.9811573 0.9824883 0.9839249 0.9840464
## [8] 0.9845243 0.9845737 0.9846936
## [1] "Optimal model variable count:"
## [1] 10The 6th model is best according to Cp. The 6th model is best according to BIC. The 8th model is best according to R-Squared.
a3<-regsubsets(Y~., data = x_data, method = "backward", nvmax = 10)
mysummary(a3)## Subset selection object
## Call: regsubsets.formula(Y ~ ., data = x_data, method = "backward",
## nvmax = 10)
## 10 Variables (and intercept)
## Forced in Forced out
## X FALSE FALSE
## `X^2` FALSE FALSE
## `X^3` FALSE FALSE
## `X^4` FALSE FALSE
## `X^5` FALSE FALSE
## `X^6` FALSE FALSE
## `X^7` FALSE FALSE
## `X^8` FALSE FALSE
## `X^9` FALSE FALSE
## `X^10` FALSE FALSE
## 1 subsets of each size up to 10
## Selection Algorithm: backward
## X `X^2` `X^3` `X^4` `X^5` `X^6` `X^7` `X^8` `X^9` `X^10`
## 1 ( 1 ) "*" " " " " " " " " " " " " " " " " " "
## 2 ( 1 ) "*" "*" " " " " " " " " " " " " " " " "
## 3 ( 1 ) "*" "*" " " " " "*" " " " " " " " " " "
## 4 ( 1 ) "*" "*" " " " " "*" " " "*" " " " " " "
## 5 ( 1 ) "*" "*" " " " " "*" " " "*" " " "*" " "
## 6 ( 1 ) "*" "*" " " " " "*" "*" "*" " " "*" " "
## 7 ( 1 ) "*" "*" " " " " "*" "*" "*" "*" "*" " "
## 8 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*" " "
## 9 ( 1 ) "*" "*" " " "*" "*" "*" "*" "*" "*" "*"
## 10 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*"
## [1] "Cps: "
## [1] 1043.992234 710.611218 122.597547 56.309625 5.568909
## [6] 7.181454 5.528537 7.337124 9.080219 11.000000
## [1] "Optimal model variable count:"
## [1] 7
## [1] "BICs: "
## [1] -153.7239 -183.9605 -311.5148 -345.2135 -385.3111 -381.1208 -380.5148
## [8] -376.1237 -371.8065 -367.2914
## [1] "Optimal model variable count:"
## [1] 5
## [1] "R-squared: "
## [1] 0.8039415 0.8616211 0.9630930 0.9748373 0.9839078 0.9839744 0.9846027
## [8] 0.9846356 0.9846798 0.9846936
## [1] "Optimal model variable count:"
## [1] 10The 7th model is best according to Cp. The 5th model is best according to BIC. The 8th model is best according to R-Squared.
The model with the most amount of variables always has the larges R-squared value becasue, unlike Cp and BIC, there is no punishment for adding cofactors in the calculation for R-squared. That being said, it is not suprising that the best model according to the R-Squared value is consistently the model with 8 variables. That model is the same between the exaustive search and backward search, which is the model that includes every X except X^3 and X^10. For forward stepwise, that model is the model that includes every X except X^6 and X^8.
According to Cp, the best model found in the exaustive search was the model with 7 variables, which included every X except X^3, X^4, and X^10. For the forward stepwise it was the model includeing 6 variables (X to the 1,2,3,5,7, and 9th power). For the backward stepwise it was the same 7 variable model found from the exaustive search.
According to BIC, the best model found in the exaustive search was the model with 5 variables (X to the 1,2,5,7, and 9th power). The same 5 variable model was found in when backward stepwise was applied. For the forward stepwise, however, the 6 variable model with X to the 1,2,3,5,7, and 9th powers was the best.
e)
#fit lasso reg. model
x_dataM = as.matrix(x_data)
grid = seq(0,10, length =100)
lasso.mod=glmnet(x_dataM[,-11],x_dataM[ ,11],alpha=1, lambda = grid)
plot(lasso.mod, label=TRUE)
#Cross-validation
set.seed(42)
#x_dataf = as.data.frame(x_data)
cv.out=cv.glmnet(x_dataM[,-11],x_dataM[ ,11],alpha=1)
plot(cv.out)
#find lambda corresponding to smallest mse
bestlam =cv.out$lambda.min
bestlam## [1] 0.168265#extract coeficients for model with this lambda
coef(cv.out, s = "lambda.min")## 11 x 1 sparse Matrix of class "dgCMatrix"
## 1
## (Intercept) 1.1188429
## X 3.0702398
## X^2 1.7706815
## X^3 0.9109737
## X^4 .
## X^5 .
## X^6 .
## X^7 .
## X^8 .
## X^9 .
## X^10 .The lambda that gives the smallest cross-validated mean-squared error is 0.168265. This lambda corresponds with the point on the cross-validation graph whose log(Lambda) = log(0.168265) = -0.77401, which is the point between the two calculated vertical dotted lines. The coefficient values can be found in the 11 x 1 spase matrix above. Notice that X, X^2, and X^3 are the only covariates with non-zero coefficients, and their coefficients are 3.07, 1.77, and 0.91, correspondingly. Remember our data was created using the real coefficient valeus of 3, 2, and 1. Thus, this model is fairly accurate in its assumptions. It overestimates the X coefficient, and underestimate both the X^2 and the X^3 coefficient. The fact that these particular covariates have non-zero coeficients for the optimal model is not suprising, since our data was generated using exclusively these three variables.
Data Generation)
# B0=1 B7=2.5
B_2 = c(1,2.5)
Y_2 = rep(B_2[1], 100)
Y_2 = Y_2 + (B_2[2]*X^7) + e
x_data_2 = data.frame(X, X^2, X^3, X^4, X^5, X^6, X^7, X^8, X^9, X^10, Y_2)
colnames(x_data) =c('X', 'X^2', 'X^3', 'X^4', 'X^5', 'X^6', 'X^7', 'X^8', 'X^9', 'X^10', 'Y_2')
x_data## X X^2 X^3 X^4 X^5
## 1 1.37095845 1.879527064 2.576754e+00 3.532622e+00 4.843078e+00
## 2 -0.56469817 0.318884025 -1.800732e-01 1.016870e-01 -5.742247e-02
## 3 0.36312841 0.131862243 4.788293e-02 1.738765e-02 6.313950e-03
## 4 0.63286260 0.400515077 2.534710e-01 1.604123e-01 1.015190e-01
## 5 0.40426832 0.163432877 6.607074e-02 2.671031e-02 1.079813e-02
## 6 -0.10612452 0.011262413 -1.195218e-03 1.268419e-04 -1.346104e-05
## 7 1.51152200 2.284698749 3.453372e+00 5.219848e+00 7.889916e+00
## 8 -0.09465904 0.008960334 -8.481766e-04 8.028758e-05 -7.599945e-06
## 9 2.01842371 4.074034289 8.223127e+00 1.659776e+01 3.350130e+01
## 10 -0.06271410 0.003933058 -2.466582e-04 1.546895e-05 -9.701211e-07
## 11 1.30486965 1.702684815 2.221782e+00 2.899136e+00 3.782994e+00
## 12 2.28664539 5.228747152 1.195629e+01 2.733980e+01 6.251642e+01
## 13 -1.38886070 1.928934047 -2.679021e+00 3.720787e+00 -5.167654e+00
## 14 -0.27878877 0.077723177 -2.166835e-02 6.040892e-03 -1.684133e-03
## 15 -0.13332134 0.017774579 -2.369731e-03 3.159356e-04 -4.212096e-05
## 16 0.63595040 0.404432909 2.571993e-01 1.635660e-01 1.040198e-01
## 17 -0.28425292 0.080799723 -2.296756e-02 6.528595e-03 -1.855772e-03
## 18 -2.65645542 7.056755403 -1.874596e+01 4.979780e+01 -1.322856e+02
## 19 -2.44046693 5.955878829 -1.453513e+01 3.547249e+01 -8.656945e+01
## 20 1.32011335 1.742699246 2.300561e+00 3.037001e+00 4.009185e+00
## 21 -0.30663859 0.094027227 -2.883238e-02 8.841119e-03 -2.711028e-03
## 22 -1.78130843 3.173059737 -5.652198e+00 1.006831e+01 -1.793476e+01
## 23 -0.17191736 0.029555577 -5.081117e-03 8.735321e-04 -1.501753e-04
## 24 1.21467470 1.475434625 1.792173e+00 2.176907e+00 2.644234e+00
## 25 1.89519346 3.591758256 6.807077e+00 1.290073e+01 2.444937e+01
## 26 -0.43046913 0.185303673 -7.976751e-02 3.433745e-02 -1.478121e-02
## 27 -0.25726938 0.066187535 -1.702803e-02 4.380790e-03 -1.127043e-03
## 28 -1.76316309 3.108744065 -5.481223e+00 9.664290e+00 -1.703972e+01
## 29 0.46009735 0.211689576 9.739781e-02 4.481248e-02 2.061810e-02
## 30 -0.63999488 0.409593441 -2.621377e-01 1.677668e-01 -1.073699e-01
## 31 0.45545012 0.207434815 9.447621e-02 4.302920e-02 1.959766e-02
## 32 0.70483734 0.496795672 3.501601e-01 2.468059e-01 1.739580e-01
## 33 1.03510352 1.071439301 1.109051e+00 1.147982e+00 1.188280e+00
## 34 -0.60892638 0.370791331 -2.257846e-01 1.374862e-01 -8.371898e-02
## 35 0.50495512 0.254979677 1.287533e-01 6.501464e-02 3.282947e-02
## 36 -1.71700868 2.948118804 -5.061946e+00 8.691404e+00 -1.492322e+01
## 37 -0.78445901 0.615375936 -4.827372e-01 3.786875e-01 -2.970649e-01
## 38 -0.85090759 0.724043734 -6.160943e-01 5.242393e-01 -4.460792e-01
## 39 -2.41420765 5.828398577 -1.407096e+01 3.397023e+01 -8.201119e+01
## 40 0.03612261 0.001304843 4.713432e-05 1.702615e-06 6.150288e-08
## 41 0.20599860 0.042435423 8.741638e-03 1.800765e-03 3.709551e-04
## 42 -0.36105730 0.130362373 -4.706829e-02 1.699435e-02 -6.135933e-03
## 43 0.75816324 0.574811492 4.358009e-01 3.304083e-01 2.505034e-01
## 44 -0.72670483 0.528099906 -3.837728e-01 2.788895e-01 -2.026704e-01
## 45 -1.36828104 1.872193017 -2.561686e+00 3.505107e+00 -4.795971e+00
## 46 0.43281803 0.187331444 8.108043e-02 3.509307e-02 1.518891e-02
## 47 -0.81139318 0.658358886 -5.341879e-01 4.334364e-01 -3.516874e-01
## 48 1.44410126 2.085428454 3.011570e+00 4.349012e+00 6.280413e+00
## 49 -0.43144620 0.186145826 -8.031191e-02 3.465027e-02 -1.494973e-02
## 50 0.65564788 0.429874147 2.818461e-01 1.847918e-01 1.211583e-01
## 51 0.32192527 0.103635876 3.336301e-02 1.074039e-02 3.457604e-03
## 52 -0.78383894 0.614403485 -4.815934e-01 3.774916e-01 -2.958926e-01
## 53 1.57572752 2.482917217 3.912401e+00 6.164878e+00 9.714168e+00
## 54 0.64289931 0.413319517 2.657228e-01 1.708330e-01 1.098284e-01
## 55 0.08976065 0.008056974 7.231992e-04 6.491482e-05 5.826797e-06
## 56 0.27655075 0.076480316 2.115069e-02 5.849239e-03 1.617611e-03
## 57 0.67928882 0.461433296 3.134465e-01 2.129207e-01 1.446346e-01
## 58 0.08983289 0.008069948 7.249467e-04 6.512405e-05 5.850282e-06
## 59 -2.99309008 8.958588246 -2.681386e+01 8.025630e+01 -2.402143e+02
## 60 0.28488295 0.081158297 2.312062e-02 6.586669e-03 1.876430e-03
## 61 -0.36723464 0.134861283 -4.952574e-02 1.818757e-02 -6.679104e-03
## 62 0.18523056 0.034310362 6.355328e-03 1.177201e-03 2.180536e-04
## 63 0.58182373 0.338518850 1.969583e-01 1.145950e-01 6.667410e-02
## 64 1.39973683 1.959263186 2.742453e+00 3.838712e+00 5.373187e+00
## 65 -0.72729206 0.528953740 -3.847039e-01 2.797921e-01 -2.034905e-01
## 66 1.30254263 1.696617308 2.209916e+00 2.878510e+00 3.749382e+00
## 67 0.33584812 0.112793960 3.788164e-02 1.272248e-02 4.272820e-03
## 68 1.03850610 1.078494917 1.120024e+00 1.163151e+00 1.207940e+00
## 69 0.92072857 0.847741096 7.805394e-01 7.186650e-01 6.616954e-01
## 70 0.72087816 0.519665326 3.746154e-01 2.700521e-01 1.946746e-01
## 71 -1.04311894 1.088097120 -1.135015e+00 1.183955e+00 -1.235006e+00
## 72 -0.09018639 0.008133584 -7.335386e-04 6.615519e-05 -5.966298e-06
## 73 0.62351816 0.388774898 2.424082e-01 1.511459e-01 9.424223e-02
## 74 -0.95352336 0.909206794 -8.669499e-01 8.266570e-01 -7.882368e-01
## 75 -0.54282881 0.294663122 -1.599516e-01 8.682636e-02 -4.713185e-02
## 76 0.58099650 0.337556930 1.961194e-01 1.139447e-01 6.620146e-02
## 77 0.76817874 0.590098573 4.533012e-01 3.482163e-01 2.674924e-01
## 78 0.46376759 0.215080376 9.974731e-02 4.625957e-02 2.145369e-02
## 79 -0.88577630 0.784599649 -6.949798e-01 6.155966e-01 -5.452809e-01
## 80 -1.09978090 1.209518025 -1.330205e+00 1.462934e+00 -1.608907e+00
## 81 1.51270701 2.288282498 3.461501e+00 5.236237e+00 7.920892e+00
## 82 0.25792144 0.066523468 1.715783e-02 4.425372e-03 1.141398e-03
## 83 0.08844023 0.007821674 6.917507e-04 6.117859e-05 5.410648e-06
## 84 -0.12089654 0.014615973 -1.767021e-03 2.136267e-04 -2.582672e-05
## 85 -1.19432890 1.426421510 -1.703616e+00 2.034678e+00 -2.430075e+00
## 86 0.61199690 0.374540203 2.292174e-01 1.402804e-01 8.585115e-02
## 87 -0.21713985 0.047149713 -1.023808e-02 2.223095e-03 -4.827226e-04
## 88 -0.18275671 0.033400014 -6.104076e-03 1.115561e-03 -2.038762e-04
## 89 0.93334633 0.871135369 8.130710e-01 7.588768e-01 7.082949e-01
## 90 0.82177311 0.675311045 5.549525e-01 4.560450e-01 3.747655e-01
## 91 1.39211638 1.937988004 2.697905e+00 3.755798e+00 5.228507e+00
## 92 -0.47617392 0.226741605 -1.079684e-01 5.141176e-02 -2.448094e-02
## 93 0.65034856 0.422953250 2.750670e-01 1.788895e-01 1.163405e-01
## 94 1.39111046 1.935188302 2.692061e+00 3.744954e+00 5.209644e+00
## 95 -1.11078888 1.233851935 -1.370549e+00 1.522391e+00 -1.691055e+00
## 96 -0.86079259 0.740963878 -6.378162e-01 5.490275e-01 -4.725988e-01
## 97 -1.13173868 1.280832442 -1.449568e+00 1.640532e+00 -1.856653e+00
## 98 -1.45921400 2.129305496 -3.107112e+00 4.533942e+00 -6.615991e+00
## 99 0.07998255 0.006397209 5.116651e-04 4.092428e-05 3.273228e-06
## 100 0.65320434 0.426675909 2.787066e-01 1.820523e-01 1.189174e-01
## X^6 X^7 X^8 X^9 X^10
## 1 6.639659e+00 9.102696e+00 1.247942e+01 1.710876e+01 2.345540e+01
## 2 3.242637e-02 -1.831111e-02 1.034025e-02 -5.839120e-03 3.297341e-03
## 3 2.292775e-03 8.325716e-04 3.023304e-04 1.097848e-04 3.986597e-05
## 4 6.424756e-02 4.065988e-02 2.573211e-02 1.628489e-02 1.030610e-02
## 5 4.365342e-03 1.764770e-03 7.134404e-04 2.884214e-04 1.165996e-04
## 6 1.428546e-06 -1.516038e-07 1.608888e-08 -1.707424e-09 1.811996e-10
## 7 1.192578e+01 1.802608e+01 2.724682e+01 4.118416e+01 6.225077e+01
## 8 7.194035e-07 -6.809804e-08 6.446095e-09 -6.101812e-10 5.775916e-11
## 9 6.761982e+01 1.364855e+02 2.754855e+02 5.560464e+02 1.122337e+03
## 10 6.084027e-08 -3.815543e-09 2.392883e-10 -1.500675e-11 9.411349e-13
## 11 4.936314e+00 6.441247e+00 8.404987e+00 1.096741e+01 1.431104e+01
## 12 1.429529e+02 3.268826e+02 7.474645e+02 1.709186e+03 3.908303e+03
## 13 7.177152e+00 -9.968064e+00 1.384425e+01 -1.922774e+01 2.670465e+01
## 14 4.695173e-04 -1.308962e-04 3.649238e-05 -1.017367e-05 2.836304e-06
## 15 5.615623e-06 -7.486824e-07 9.981533e-08 -1.330751e-08 1.774176e-09
## 16 6.615146e-02 4.206905e-02 2.675383e-02 1.701411e-02 1.082013e-02
## 17 5.275087e-04 -1.499459e-04 4.262256e-05 -1.211559e-05 3.443891e-06
## 18 3.514109e+02 -9.335073e+02 2.479821e+03 -6.587533e+03 1.749949e+04
## 19 2.112699e+02 -5.155971e+02 1.258298e+03 -3.070834e+03 7.494269e+03
## 20 5.292579e+00 6.986804e+00 9.223373e+00 1.217590e+01 1.607357e+01
## 21 8.313060e-04 -2.549105e-04 7.816539e-05 -2.396853e-05 7.349675e-06
## 22 3.194734e+01 -5.690807e+01 1.013708e+02 -1.805727e+02 3.216557e+02
## 23 2.581775e-05 -4.438519e-06 7.630584e-07 -1.311830e-07 2.255263e-08
## 24 3.211884e+00 3.901395e+00 4.738926e+00 5.756253e+00 6.991975e+00
## 25 4.633629e+01 8.781624e+01 1.664288e+02 3.154147e+02 5.977719e+02
## 26 6.362856e-03 -2.739013e-03 1.179061e-03 -5.075492e-04 2.184843e-04
## 27 2.899537e-04 -7.459620e-05 1.919132e-05 -4.937339e-06 1.270226e-06
## 28 3.004380e+01 -5.297212e+01 9.339849e+01 -1.646768e+02 2.903520e+02
## 29 9.486334e-03 4.364637e-03 2.008158e-03 9.239482e-04 4.251061e-04
## 30 6.871618e-02 -4.397800e-02 2.814569e-02 -1.801310e-02 1.152829e-02
## 31 8.925755e-03 4.065236e-03 1.851512e-03 8.432715e-04 3.840681e-04
## 32 1.226121e-01 8.642160e-02 6.091317e-02 4.293388e-02 3.026140e-02
## 33 1.229993e+00 1.273170e+00 1.317863e+00 1.364125e+00 1.412010e+00
## 34 5.097870e-02 -3.104227e-02 1.890246e-02 -1.151021e-02 7.008868e-03
## 35 1.657741e-02 8.370848e-03 4.226903e-03 2.134396e-03 1.077774e-03
## 36 2.562329e+01 -4.399542e+01 7.554051e+01 -1.297037e+02 2.227024e+02
## 37 2.330352e-01 -1.828066e-01 1.434043e-01 -1.124948e-01 8.824753e-02
## 38 3.795722e-01 -3.229809e-01 2.748269e-01 -2.338523e-01 1.989867e-01
## 39 1.979920e+02 -4.779939e+02 1.153977e+03 -2.785939e+03 6.725835e+03
## 40 2.221644e-09 8.025158e-11 2.898896e-12 1.047157e-13 3.782604e-15
## 41 7.641623e-05 1.574164e-05 3.242755e-06 6.680030e-07 1.376077e-07
## 42 2.215424e-03 -7.998948e-04 2.888079e-04 -1.042762e-04 3.764968e-05
## 43 1.899225e-01 1.439922e-01 1.091696e-01 8.276839e-02 6.275195e-02
## 44 1.472815e-01 -1.070302e-01 7.777936e-02 -5.652264e-02 4.107527e-02
## 45 6.562236e+00 -8.978983e+00 1.228577e+01 -1.681039e+01 2.300134e+01
## 46 6.574035e-03 2.845361e-03 1.231524e-03 5.330256e-04 2.307031e-04
## 47 2.853567e-01 -2.315365e-01 1.878671e-01 -1.524341e-01 1.236840e-01
## 48 9.069553e+00 1.309735e+01 1.891390e+01 2.731359e+01 3.944359e+01
## 49 6.450003e-03 -2.782829e-03 1.200641e-03 -5.180120e-04 2.234943e-04
## 50 7.943721e-02 5.208284e-02 3.414800e-02 2.238907e-02 1.467934e-02
## 51 1.113090e-03 3.583319e-04 1.153561e-04 3.713604e-05 1.195503e-05
## 52 2.319322e-01 -1.817975e-01 1.424999e-01 -1.116970e-01 8.755246e-02
## 53 1.530688e+01 2.411947e+01 3.800572e+01 5.988666e+01 9.436506e+01
## 54 7.060862e-02 4.539423e-02 2.918392e-02 1.876232e-02 1.206228e-02
## 55 5.230170e-07 4.694635e-08 4.213934e-09 3.782455e-10 3.395156e-11
## 56 4.473516e-04 1.237154e-04 3.421359e-05 9.461795e-06 2.616666e-06
## 57 9.824869e-02 6.673924e-02 4.533522e-02 3.079571e-02 2.091918e-02
## 58 5.255477e-07 4.721147e-08 4.241142e-09 3.809941e-10 3.422580e-11
## 59 7.189832e+02 -2.151981e+03 6.441074e+03 -1.927872e+04 5.770293e+04
## 60 5.345629e-04 1.522878e-04 4.338421e-05 1.235942e-05 3.520989e-06
## 61 2.452798e-03 -9.007526e-04 3.307875e-04 -1.214766e-04 4.461043e-05
## 62 4.039019e-05 7.481498e-06 1.385802e-06 2.566929e-07 4.754737e-08
## 63 3.879257e-02 2.257044e-02 1.313202e-02 7.640519e-03 4.445435e-03
## 64 7.521048e+00 1.052749e+01 1.473571e+01 2.062612e+01 2.887114e+01
## 65 1.479971e-01 -1.076371e-01 7.828360e-02 -5.693504e-02 4.140840e-02
## 66 4.883730e+00 6.361267e+00 8.285821e+00 1.079264e+01 1.405787e+01
## 67 1.435019e-03 4.819483e-04 1.618614e-04 5.436086e-05 1.825699e-05
## 68 1.254453e+00 1.302757e+00 1.352921e+00 1.405017e+00 1.459118e+00
## 69 6.092418e-01 5.609464e-01 5.164793e-01 4.755373e-01 4.378408e-01
## 70 1.403367e-01 1.011657e-01 7.292811e-02 5.257228e-02 3.789821e-02
## 71 1.288258e+00 -1.343807e+00 1.401750e+00 -1.462192e+00 1.525240e+00
## 72 5.380788e-07 -4.852739e-08 4.376510e-09 -3.947016e-10 3.559671e-11
## 73 5.876174e-02 3.663901e-02 2.284509e-02 1.424433e-02 8.881597e-03
## 74 7.516022e-01 -7.166702e-01 6.833618e-01 -6.516014e-01 6.213172e-01
## 75 2.558452e-02 -1.388802e-02 7.538816e-03 -4.092287e-03 2.221411e-03
## 76 3.846282e-02 2.234676e-02 1.298339e-02 7.543304e-03 4.382633e-03
## 77 2.054820e-01 1.578469e-01 1.212546e-01 9.314521e-02 7.155217e-02
## 78 9.949525e-03 4.614267e-03 2.139948e-03 9.924384e-04 4.602607e-04
## 79 4.829969e-01 -4.278272e-01 3.789592e-01 -3.356731e-01 2.973312e-01
## 80 1.769445e+00 -1.946002e+00 2.140175e+00 -2.353724e+00 2.588581e+00
## 81 1.198199e+01 1.812524e+01 2.741818e+01 4.147567e+01 6.274053e+01
## 82 2.943911e-04 7.592977e-05 1.958392e-05 5.051112e-06 1.302790e-06
## 83 4.785190e-07 4.232033e-08 3.742819e-09 3.310158e-10 2.927511e-11
## 84 3.122361e-06 -3.774827e-07 4.563635e-08 -5.517277e-09 6.670197e-10
## 85 2.902309e+00 -3.466311e+00 4.139916e+00 -4.944421e+00 5.905265e+00
## 86 5.254064e-02 3.215471e-02 1.967858e-02 1.204323e-02 7.370420e-03
## 87 1.048183e-04 -2.276023e-05 4.942153e-06 -1.073138e-06 2.330211e-07
## 88 3.725975e-05 -6.809469e-06 1.244476e-06 -2.274364e-07 4.156552e-08
## 89 6.610844e-01 6.170207e-01 5.758940e-01 5.375086e-01 5.016817e-01
## 90 3.079722e-01 2.530833e-01 2.079770e-01 1.709099e-01 1.404492e-01
## 91 7.278691e+00 1.013278e+01 1.410601e+01 1.963721e+01 2.733729e+01
## 92 1.165718e-02 -5.550847e-03 2.643169e-03 -1.258608e-03 5.993163e-04
## 93 7.566188e-02 4.920659e-02 3.200144e-02 2.081209e-02 1.353511e-02
## 94 7.247191e+00 1.008164e+01 1.402468e+01 1.950988e+01 2.714039e+01
## 95 1.878405e+00 -2.086511e+00 2.317673e+00 -2.574446e+00 2.859665e+00
## 96 4.068095e-01 -3.501786e-01 3.014312e-01 -2.594697e-01 2.233496e-01
## 97 2.101246e+00 -2.378062e+00 2.691344e+00 -3.045899e+00 3.447161e+00
## 98 9.654147e+00 -1.408747e+01 2.055663e+01 -2.999652e+01 4.377134e+01
## 99 2.618012e-07 2.093953e-08 1.674797e-09 1.339545e-10 1.071402e-11
## 100 7.767734e-02 5.073918e-02 3.314305e-02 2.164919e-02 1.414134e-02
## Y_2
## 1 12.64964835
## 2 0.80835140
## 3 1.39778400
## 4 5.80157088
## 5 1.93896805
## 6 0.80846987
## 7 13.13508002
## 8 0.61074520
## 9 23.61466017
## 10 0.93863812
## 11 10.51666779
## 12 30.38179379
## 13 -2.47316994
## 14 -0.20680543
## 15 -1.02788366
## 16 3.59158255
## 17 -0.22677713
## 18 -8.89992060
## 19 -10.30688467
## 20 10.88355528
## 21 -1.25431877
## 22 -5.12043964
## 23 0.66298036
## 24 8.39042732
## 25 20.67435104
## 26 -0.42882644
## 27 -0.27013271
## 28 -5.57790175
## 29 1.67632108
## 30 -0.18341901
## 31 3.44331681
## 32 3.96538614
## 33 7.35730265
## 34 0.81190856
## 35 4.59343376
## 36 -4.41384777
## 37 -0.72268191
## 38 0.48076877
## 39 -9.12651981
## 40 1.05855516
## 41 1.62550099
## 42 -0.75719445
## 43 4.41522963
## 44 -0.53713230
## 45 -2.33601217
## 46 3.86758341
## 47 -1.13264251
## 48 12.08156152
## 49 0.69450371
## 50 3.05216961
## 51 2.16571208
## 52 -2.15584805
## 53 15.77258753
## 54 3.74741408
## 55 0.81827376
## 56 0.76551123
## 57 4.26641748
## 58 0.48608132
## 59 -17.40944773
## 60 3.32776132
## 61 -0.05703297
## 62 0.55888536
## 63 3.78267406
## 64 11.49745127
## 65 0.08134099
## 66 11.94320081
## 67 1.27832141
## 68 7.84718198
## 69 6.32310540
## 70 5.47214611
## 71 -1.31795543
## 72 1.58159354
## 73 2.14545663
## 74 0.78035252
## 75 0.66566615
## 76 3.46344676
## 77 3.48902741
## 78 3.56421953
## 79 -0.29991550
## 80 -1.21686709
## 81 13.72764289
## 82 1.33986011
## 83 1.65046252
## 84 0.95942965
## 85 -1.71301947
## 86 2.47805189
## 87 1.13339063
## 88 1.06662245
## 89 5.51907413
## 90 3.77630572
## 91 11.95518855
## 92 -0.42809498
## 93 4.32463092
## 94 10.44176619
## 95 -2.19438222
## 96 0.34750864
## 97 -0.87934387
## 98 -1.63965586
## 99 3.06848219
## 100 4.22049282x_data_2## X X.2 X.3 X.4 X.5
## 1 1.37095845 1.879527064 2.576754e+00 3.532622e+00 4.843078e+00
## 2 -0.56469817 0.318884025 -1.800732e-01 1.016870e-01 -5.742247e-02
## 3 0.36312841 0.131862243 4.788293e-02 1.738765e-02 6.313950e-03
## 4 0.63286260 0.400515077 2.534710e-01 1.604123e-01 1.015190e-01
## 5 0.40426832 0.163432877 6.607074e-02 2.671031e-02 1.079813e-02
## 6 -0.10612452 0.011262413 -1.195218e-03 1.268419e-04 -1.346104e-05
## 7 1.51152200 2.284698749 3.453372e+00 5.219848e+00 7.889916e+00
## 8 -0.09465904 0.008960334 -8.481766e-04 8.028758e-05 -7.599945e-06
## 9 2.01842371 4.074034289 8.223127e+00 1.659776e+01 3.350130e+01
## 10 -0.06271410 0.003933058 -2.466582e-04 1.546895e-05 -9.701211e-07
## 11 1.30486965 1.702684815 2.221782e+00 2.899136e+00 3.782994e+00
## 12 2.28664539 5.228747152 1.195629e+01 2.733980e+01 6.251642e+01
## 13 -1.38886070 1.928934047 -2.679021e+00 3.720787e+00 -5.167654e+00
## 14 -0.27878877 0.077723177 -2.166835e-02 6.040892e-03 -1.684133e-03
## 15 -0.13332134 0.017774579 -2.369731e-03 3.159356e-04 -4.212096e-05
## 16 0.63595040 0.404432909 2.571993e-01 1.635660e-01 1.040198e-01
## 17 -0.28425292 0.080799723 -2.296756e-02 6.528595e-03 -1.855772e-03
## 18 -2.65645542 7.056755403 -1.874596e+01 4.979780e+01 -1.322856e+02
## 19 -2.44046693 5.955878829 -1.453513e+01 3.547249e+01 -8.656945e+01
## 20 1.32011335 1.742699246 2.300561e+00 3.037001e+00 4.009185e+00
## 21 -0.30663859 0.094027227 -2.883238e-02 8.841119e-03 -2.711028e-03
## 22 -1.78130843 3.173059737 -5.652198e+00 1.006831e+01 -1.793476e+01
## 23 -0.17191736 0.029555577 -5.081117e-03 8.735321e-04 -1.501753e-04
## 24 1.21467470 1.475434625 1.792173e+00 2.176907e+00 2.644234e+00
## 25 1.89519346 3.591758256 6.807077e+00 1.290073e+01 2.444937e+01
## 26 -0.43046913 0.185303673 -7.976751e-02 3.433745e-02 -1.478121e-02
## 27 -0.25726938 0.066187535 -1.702803e-02 4.380790e-03 -1.127043e-03
## 28 -1.76316309 3.108744065 -5.481223e+00 9.664290e+00 -1.703972e+01
## 29 0.46009735 0.211689576 9.739781e-02 4.481248e-02 2.061810e-02
## 30 -0.63999488 0.409593441 -2.621377e-01 1.677668e-01 -1.073699e-01
## 31 0.45545012 0.207434815 9.447621e-02 4.302920e-02 1.959766e-02
## 32 0.70483734 0.496795672 3.501601e-01 2.468059e-01 1.739580e-01
## 33 1.03510352 1.071439301 1.109051e+00 1.147982e+00 1.188280e+00
## 34 -0.60892638 0.370791331 -2.257846e-01 1.374862e-01 -8.371898e-02
## 35 0.50495512 0.254979677 1.287533e-01 6.501464e-02 3.282947e-02
## 36 -1.71700868 2.948118804 -5.061946e+00 8.691404e+00 -1.492322e+01
## 37 -0.78445901 0.615375936 -4.827372e-01 3.786875e-01 -2.970649e-01
## 38 -0.85090759 0.724043734 -6.160943e-01 5.242393e-01 -4.460792e-01
## 39 -2.41420765 5.828398577 -1.407096e+01 3.397023e+01 -8.201119e+01
## 40 0.03612261 0.001304843 4.713432e-05 1.702615e-06 6.150288e-08
## 41 0.20599860 0.042435423 8.741638e-03 1.800765e-03 3.709551e-04
## 42 -0.36105730 0.130362373 -4.706829e-02 1.699435e-02 -6.135933e-03
## 43 0.75816324 0.574811492 4.358009e-01 3.304083e-01 2.505034e-01
## 44 -0.72670483 0.528099906 -3.837728e-01 2.788895e-01 -2.026704e-01
## 45 -1.36828104 1.872193017 -2.561686e+00 3.505107e+00 -4.795971e+00
## 46 0.43281803 0.187331444 8.108043e-02 3.509307e-02 1.518891e-02
## 47 -0.81139318 0.658358886 -5.341879e-01 4.334364e-01 -3.516874e-01
## 48 1.44410126 2.085428454 3.011570e+00 4.349012e+00 6.280413e+00
## 49 -0.43144620 0.186145826 -8.031191e-02 3.465027e-02 -1.494973e-02
## 50 0.65564788 0.429874147 2.818461e-01 1.847918e-01 1.211583e-01
## 51 0.32192527 0.103635876 3.336301e-02 1.074039e-02 3.457604e-03
## 52 -0.78383894 0.614403485 -4.815934e-01 3.774916e-01 -2.958926e-01
## 53 1.57572752 2.482917217 3.912401e+00 6.164878e+00 9.714168e+00
## 54 0.64289931 0.413319517 2.657228e-01 1.708330e-01 1.098284e-01
## 55 0.08976065 0.008056974 7.231992e-04 6.491482e-05 5.826797e-06
## 56 0.27655075 0.076480316 2.115069e-02 5.849239e-03 1.617611e-03
## 57 0.67928882 0.461433296 3.134465e-01 2.129207e-01 1.446346e-01
## 58 0.08983289 0.008069948 7.249467e-04 6.512405e-05 5.850282e-06
## 59 -2.99309008 8.958588246 -2.681386e+01 8.025630e+01 -2.402143e+02
## 60 0.28488295 0.081158297 2.312062e-02 6.586669e-03 1.876430e-03
## 61 -0.36723464 0.134861283 -4.952574e-02 1.818757e-02 -6.679104e-03
## 62 0.18523056 0.034310362 6.355328e-03 1.177201e-03 2.180536e-04
## 63 0.58182373 0.338518850 1.969583e-01 1.145950e-01 6.667410e-02
## 64 1.39973683 1.959263186 2.742453e+00 3.838712e+00 5.373187e+00
## 65 -0.72729206 0.528953740 -3.847039e-01 2.797921e-01 -2.034905e-01
## 66 1.30254263 1.696617308 2.209916e+00 2.878510e+00 3.749382e+00
## 67 0.33584812 0.112793960 3.788164e-02 1.272248e-02 4.272820e-03
## 68 1.03850610 1.078494917 1.120024e+00 1.163151e+00 1.207940e+00
## 69 0.92072857 0.847741096 7.805394e-01 7.186650e-01 6.616954e-01
## 70 0.72087816 0.519665326 3.746154e-01 2.700521e-01 1.946746e-01
## 71 -1.04311894 1.088097120 -1.135015e+00 1.183955e+00 -1.235006e+00
## 72 -0.09018639 0.008133584 -7.335386e-04 6.615519e-05 -5.966298e-06
## 73 0.62351816 0.388774898 2.424082e-01 1.511459e-01 9.424223e-02
## 74 -0.95352336 0.909206794 -8.669499e-01 8.266570e-01 -7.882368e-01
## 75 -0.54282881 0.294663122 -1.599516e-01 8.682636e-02 -4.713185e-02
## 76 0.58099650 0.337556930 1.961194e-01 1.139447e-01 6.620146e-02
## 77 0.76817874 0.590098573 4.533012e-01 3.482163e-01 2.674924e-01
## 78 0.46376759 0.215080376 9.974731e-02 4.625957e-02 2.145369e-02
## 79 -0.88577630 0.784599649 -6.949798e-01 6.155966e-01 -5.452809e-01
## 80 -1.09978090 1.209518025 -1.330205e+00 1.462934e+00 -1.608907e+00
## 81 1.51270701 2.288282498 3.461501e+00 5.236237e+00 7.920892e+00
## 82 0.25792144 0.066523468 1.715783e-02 4.425372e-03 1.141398e-03
## 83 0.08844023 0.007821674 6.917507e-04 6.117859e-05 5.410648e-06
## 84 -0.12089654 0.014615973 -1.767021e-03 2.136267e-04 -2.582672e-05
## 85 -1.19432890 1.426421510 -1.703616e+00 2.034678e+00 -2.430075e+00
## 86 0.61199690 0.374540203 2.292174e-01 1.402804e-01 8.585115e-02
## 87 -0.21713985 0.047149713 -1.023808e-02 2.223095e-03 -4.827226e-04
## 88 -0.18275671 0.033400014 -6.104076e-03 1.115561e-03 -2.038762e-04
## 89 0.93334633 0.871135369 8.130710e-01 7.588768e-01 7.082949e-01
## 90 0.82177311 0.675311045 5.549525e-01 4.560450e-01 3.747655e-01
## 91 1.39211638 1.937988004 2.697905e+00 3.755798e+00 5.228507e+00
## 92 -0.47617392 0.226741605 -1.079684e-01 5.141176e-02 -2.448094e-02
## 93 0.65034856 0.422953250 2.750670e-01 1.788895e-01 1.163405e-01
## 94 1.39111046 1.935188302 2.692061e+00 3.744954e+00 5.209644e+00
## 95 -1.11078888 1.233851935 -1.370549e+00 1.522391e+00 -1.691055e+00
## 96 -0.86079259 0.740963878 -6.378162e-01 5.490275e-01 -4.725988e-01
## 97 -1.13173868 1.280832442 -1.449568e+00 1.640532e+00 -1.856653e+00
## 98 -1.45921400 2.129305496 -3.107112e+00 4.533942e+00 -6.615991e+00
## 99 0.07998255 0.006397209 5.116651e-04 4.092428e-05 3.273228e-06
## 100 0.65320434 0.426675909 2.787066e-01 1.820523e-01 1.189174e-01
## X.6 X.7 X.8 X.9 X.10
## 1 6.639659e+00 9.102696e+00 1.247942e+01 1.710876e+01 2.345540e+01
## 2 3.242637e-02 -1.831111e-02 1.034025e-02 -5.839120e-03 3.297341e-03
## 3 2.292775e-03 8.325716e-04 3.023304e-04 1.097848e-04 3.986597e-05
## 4 6.424756e-02 4.065988e-02 2.573211e-02 1.628489e-02 1.030610e-02
## 5 4.365342e-03 1.764770e-03 7.134404e-04 2.884214e-04 1.165996e-04
## 6 1.428546e-06 -1.516038e-07 1.608888e-08 -1.707424e-09 1.811996e-10
## 7 1.192578e+01 1.802608e+01 2.724682e+01 4.118416e+01 6.225077e+01
## 8 7.194035e-07 -6.809804e-08 6.446095e-09 -6.101812e-10 5.775916e-11
## 9 6.761982e+01 1.364855e+02 2.754855e+02 5.560464e+02 1.122337e+03
## 10 6.084027e-08 -3.815543e-09 2.392883e-10 -1.500675e-11 9.411349e-13
## 11 4.936314e+00 6.441247e+00 8.404987e+00 1.096741e+01 1.431104e+01
## 12 1.429529e+02 3.268826e+02 7.474645e+02 1.709186e+03 3.908303e+03
## 13 7.177152e+00 -9.968064e+00 1.384425e+01 -1.922774e+01 2.670465e+01
## 14 4.695173e-04 -1.308962e-04 3.649238e-05 -1.017367e-05 2.836304e-06
## 15 5.615623e-06 -7.486824e-07 9.981533e-08 -1.330751e-08 1.774176e-09
## 16 6.615146e-02 4.206905e-02 2.675383e-02 1.701411e-02 1.082013e-02
## 17 5.275087e-04 -1.499459e-04 4.262256e-05 -1.211559e-05 3.443891e-06
## 18 3.514109e+02 -9.335073e+02 2.479821e+03 -6.587533e+03 1.749949e+04
## 19 2.112699e+02 -5.155971e+02 1.258298e+03 -3.070834e+03 7.494269e+03
## 20 5.292579e+00 6.986804e+00 9.223373e+00 1.217590e+01 1.607357e+01
## 21 8.313060e-04 -2.549105e-04 7.816539e-05 -2.396853e-05 7.349675e-06
## 22 3.194734e+01 -5.690807e+01 1.013708e+02 -1.805727e+02 3.216557e+02
## 23 2.581775e-05 -4.438519e-06 7.630584e-07 -1.311830e-07 2.255263e-08
## 24 3.211884e+00 3.901395e+00 4.738926e+00 5.756253e+00 6.991975e+00
## 25 4.633629e+01 8.781624e+01 1.664288e+02 3.154147e+02 5.977719e+02
## 26 6.362856e-03 -2.739013e-03 1.179061e-03 -5.075492e-04 2.184843e-04
## 27 2.899537e-04 -7.459620e-05 1.919132e-05 -4.937339e-06 1.270226e-06
## 28 3.004380e+01 -5.297212e+01 9.339849e+01 -1.646768e+02 2.903520e+02
## 29 9.486334e-03 4.364637e-03 2.008158e-03 9.239482e-04 4.251061e-04
## 30 6.871618e-02 -4.397800e-02 2.814569e-02 -1.801310e-02 1.152829e-02
## 31 8.925755e-03 4.065236e-03 1.851512e-03 8.432715e-04 3.840681e-04
## 32 1.226121e-01 8.642160e-02 6.091317e-02 4.293388e-02 3.026140e-02
## 33 1.229993e+00 1.273170e+00 1.317863e+00 1.364125e+00 1.412010e+00
## 34 5.097870e-02 -3.104227e-02 1.890246e-02 -1.151021e-02 7.008868e-03
## 35 1.657741e-02 8.370848e-03 4.226903e-03 2.134396e-03 1.077774e-03
## 36 2.562329e+01 -4.399542e+01 7.554051e+01 -1.297037e+02 2.227024e+02
## 37 2.330352e-01 -1.828066e-01 1.434043e-01 -1.124948e-01 8.824753e-02
## 38 3.795722e-01 -3.229809e-01 2.748269e-01 -2.338523e-01 1.989867e-01
## 39 1.979920e+02 -4.779939e+02 1.153977e+03 -2.785939e+03 6.725835e+03
## 40 2.221644e-09 8.025158e-11 2.898896e-12 1.047157e-13 3.782604e-15
## 41 7.641623e-05 1.574164e-05 3.242755e-06 6.680030e-07 1.376077e-07
## 42 2.215424e-03 -7.998948e-04 2.888079e-04 -1.042762e-04 3.764968e-05
## 43 1.899225e-01 1.439922e-01 1.091696e-01 8.276839e-02 6.275195e-02
## 44 1.472815e-01 -1.070302e-01 7.777936e-02 -5.652264e-02 4.107527e-02
## 45 6.562236e+00 -8.978983e+00 1.228577e+01 -1.681039e+01 2.300134e+01
## 46 6.574035e-03 2.845361e-03 1.231524e-03 5.330256e-04 2.307031e-04
## 47 2.853567e-01 -2.315365e-01 1.878671e-01 -1.524341e-01 1.236840e-01
## 48 9.069553e+00 1.309735e+01 1.891390e+01 2.731359e+01 3.944359e+01
## 49 6.450003e-03 -2.782829e-03 1.200641e-03 -5.180120e-04 2.234943e-04
## 50 7.943721e-02 5.208284e-02 3.414800e-02 2.238907e-02 1.467934e-02
## 51 1.113090e-03 3.583319e-04 1.153561e-04 3.713604e-05 1.195503e-05
## 52 2.319322e-01 -1.817975e-01 1.424999e-01 -1.116970e-01 8.755246e-02
## 53 1.530688e+01 2.411947e+01 3.800572e+01 5.988666e+01 9.436506e+01
## 54 7.060862e-02 4.539423e-02 2.918392e-02 1.876232e-02 1.206228e-02
## 55 5.230170e-07 4.694635e-08 4.213934e-09 3.782455e-10 3.395156e-11
## 56 4.473516e-04 1.237154e-04 3.421359e-05 9.461795e-06 2.616666e-06
## 57 9.824869e-02 6.673924e-02 4.533522e-02 3.079571e-02 2.091918e-02
## 58 5.255477e-07 4.721147e-08 4.241142e-09 3.809941e-10 3.422580e-11
## 59 7.189832e+02 -2.151981e+03 6.441074e+03 -1.927872e+04 5.770293e+04
## 60 5.345629e-04 1.522878e-04 4.338421e-05 1.235942e-05 3.520989e-06
## 61 2.452798e-03 -9.007526e-04 3.307875e-04 -1.214766e-04 4.461043e-05
## 62 4.039019e-05 7.481498e-06 1.385802e-06 2.566929e-07 4.754737e-08
## 63 3.879257e-02 2.257044e-02 1.313202e-02 7.640519e-03 4.445435e-03
## 64 7.521048e+00 1.052749e+01 1.473571e+01 2.062612e+01 2.887114e+01
## 65 1.479971e-01 -1.076371e-01 7.828360e-02 -5.693504e-02 4.140840e-02
## 66 4.883730e+00 6.361267e+00 8.285821e+00 1.079264e+01 1.405787e+01
## 67 1.435019e-03 4.819483e-04 1.618614e-04 5.436086e-05 1.825699e-05
## 68 1.254453e+00 1.302757e+00 1.352921e+00 1.405017e+00 1.459118e+00
## 69 6.092418e-01 5.609464e-01 5.164793e-01 4.755373e-01 4.378408e-01
## 70 1.403367e-01 1.011657e-01 7.292811e-02 5.257228e-02 3.789821e-02
## 71 1.288258e+00 -1.343807e+00 1.401750e+00 -1.462192e+00 1.525240e+00
## 72 5.380788e-07 -4.852739e-08 4.376510e-09 -3.947016e-10 3.559671e-11
## 73 5.876174e-02 3.663901e-02 2.284509e-02 1.424433e-02 8.881597e-03
## 74 7.516022e-01 -7.166702e-01 6.833618e-01 -6.516014e-01 6.213172e-01
## 75 2.558452e-02 -1.388802e-02 7.538816e-03 -4.092287e-03 2.221411e-03
## 76 3.846282e-02 2.234676e-02 1.298339e-02 7.543304e-03 4.382633e-03
## 77 2.054820e-01 1.578469e-01 1.212546e-01 9.314521e-02 7.155217e-02
## 78 9.949525e-03 4.614267e-03 2.139948e-03 9.924384e-04 4.602607e-04
## 79 4.829969e-01 -4.278272e-01 3.789592e-01 -3.356731e-01 2.973312e-01
## 80 1.769445e+00 -1.946002e+00 2.140175e+00 -2.353724e+00 2.588581e+00
## 81 1.198199e+01 1.812524e+01 2.741818e+01 4.147567e+01 6.274053e+01
## 82 2.943911e-04 7.592977e-05 1.958392e-05 5.051112e-06 1.302790e-06
## 83 4.785190e-07 4.232033e-08 3.742819e-09 3.310158e-10 2.927511e-11
## 84 3.122361e-06 -3.774827e-07 4.563635e-08 -5.517277e-09 6.670197e-10
## 85 2.902309e+00 -3.466311e+00 4.139916e+00 -4.944421e+00 5.905265e+00
## 86 5.254064e-02 3.215471e-02 1.967858e-02 1.204323e-02 7.370420e-03
## 87 1.048183e-04 -2.276023e-05 4.942153e-06 -1.073138e-06 2.330211e-07
## 88 3.725975e-05 -6.809469e-06 1.244476e-06 -2.274364e-07 4.156552e-08
## 89 6.610844e-01 6.170207e-01 5.758940e-01 5.375086e-01 5.016817e-01
## 90 3.079722e-01 2.530833e-01 2.079770e-01 1.709099e-01 1.404492e-01
## 91 7.278691e+00 1.013278e+01 1.410601e+01 1.963721e+01 2.733729e+01
## 92 1.165718e-02 -5.550847e-03 2.643169e-03 -1.258608e-03 5.993163e-04
## 93 7.566188e-02 4.920659e-02 3.200144e-02 2.081209e-02 1.353511e-02
## 94 7.247191e+00 1.008164e+01 1.402468e+01 1.950988e+01 2.714039e+01
## 95 1.878405e+00 -2.086511e+00 2.317673e+00 -2.574446e+00 2.859665e+00
## 96 4.068095e-01 -3.501786e-01 3.014312e-01 -2.594697e-01 2.233496e-01
## 97 2.101246e+00 -2.378062e+00 2.691344e+00 -3.045899e+00 3.447161e+00
## 98 9.654147e+00 -1.408747e+01 2.055663e+01 -2.999652e+01 4.377134e+01
## 99 2.618012e-07 2.093953e-08 1.674797e-09 1.339545e-10 1.071402e-11
## 100 7.767734e-02 5.073918e-02 3.314305e-02 2.164919e-02 1.414134e-02
## Y_2
## 1 2.495771e+01
## 2 1.998973e+00
## 3 -1.127218e-03
## 4 2.950132e+00
## 5 3.376385e-01
## 6 1.105513e+00
## 7 4.564295e+01
## 8 8.776497e-01
## 9 3.424018e+02
## 10 1.119161e+00
## 11 1.707802e+01
## 12 8.183145e+02
## 13 -2.440560e+01
## 14 4.954556e-01
## 15 -6.611010e-01
## 16 7.228389e-01
## 17 4.869749e-01
## 18 -2.330066e+03
## 19 -1.289355e+03
## 20 1.860427e+01
## 21 -4.942623e-01
## 22 -1.427406e+02
## 23 1.124691e+00
## 24 9.756848e+00
## 25 2.205388e+02
## 26 5.648936e-01
## 27 3.861419e-01
## 28 -1.334550e+02
## 29 -2.138364e-01
## 30 1.069571e+00
## 31 1.577784e+00
## 32 7.231767e-01
## 33 4.182989e+00
## 34 2.045284e+00
## 35 2.460783e+00
## 36 -1.100857e+02
## 37 4.256640e-01
## 38 1.394046e+00
## 39 -1.194454e+03
## 40 9.475305e-01
## 41 9.139321e-01
## 42 1.103212e-01
## 43 9.152966e-01
## 44 7.029796e-01
## 45 -2.186133e+01
## 46 2.120499e+00
## 47 -5.983408e-02
## 48 3.331021e+01
## 49 1.689906e+00
## 50 7.383868e-02
## 51 9.601974e-01
## 52 -1.006039e+00
## 53 6.246586e+01
## 54 8.398399e-01
## 55 5.321548e-01
## 56 -2.379430e-01
## 57 1.159086e+00
## 58 1.997179e-01
## 59 -5.379487e+03
## 60 2.288056e+00
## 61 8.222222e-01
## 62 -7.176368e-02
## 63 1.219633e+00
## 64 2.695598e+01
## 65 1.320921e+00
## 66 1.833559e+01
## 67 8.512360e-03
## 68 4.711542e+00
## 69 2.487264e+00
## 70 2.148480e+00
## 71 -2.589295e+00
## 72 1.836619e+00
## 73 -6.534583e-01
## 74 8.977834e-01
## 75 1.830058e+00
## 76 9.050909e-01
## 77 -5.438995e-02
## 78 1.654544e+00
## 79 4.136259e-01
## 80 -3.871360e+00
## 81 4.646455e+01
## 82 4.160809e-01
## 83 1.368807e+00
## 84 1.294653e+00
## 85 -7.945038e+00
## 86 -2.558499e-01
## 87 1.700692e+00
## 88 1.554180e+00
## 89 1.706245e+00
## 90 3.812008e-02
## 91 2.653692e+01
## 92 6.410349e-01
## 93 1.375628e+00
## 94 2.491010e+01
## 95 -5.175448e+00
## 96 1.210328e+00
## 97 -4.541379e+00
## 98 -3.363218e+01
## 99 2.815228e+00
## 100 1.255669e+00Subset Selection)
#Exhaustive
b<-regsubsets(Y_2~., data = x_data_2, nvmax = 10)
mysummary(b)## Subset selection object
## Call: regsubsets.formula(Y_2 ~ ., data = x_data_2, nvmax = 10)
## 10 Variables (and intercept)
## Forced in Forced out
## X FALSE FALSE
## X.2 FALSE FALSE
## X.3 FALSE FALSE
## X.4 FALSE FALSE
## X.5 FALSE FALSE
## X.6 FALSE FALSE
## X.7 FALSE FALSE
## X.8 FALSE FALSE
## X.9 FALSE FALSE
## X.10 FALSE FALSE
## 1 subsets of each size up to 10
## Selection Algorithm: exhaustive
## X X.2 X.3 X.4 X.5 X.6 X.7 X.8 X.9 X.10
## 1 ( 1 ) " " " " " " " " " " " " "*" " " " " " "
## 2 ( 1 ) " " " " "*" " " " " " " "*" " " " " " "
## 3 ( 1 ) " " " " " " " " "*" " " "*" " " "*" " "
## 4 ( 1 ) " " " " "*" " " "*" " " "*" " " "*" " "
## 5 ( 1 ) " " " " "*" "*" "*" " " "*" " " "*" " "
## 6 ( 1 ) " " " " "*" " " "*" " " "*" "*" "*" "*"
## 7 ( 1 ) "*" " " " " "*" "*" "*" "*" "*" " " "*"
## 8 ( 1 ) " " " " "*" "*" "*" "*" "*" "*" "*" "*"
## 9 ( 1 ) " " "*" "*" "*" "*" "*" "*" "*" "*" "*"
## 10 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*"
## [1] "Cps: "
## [1] 17.503967 18.089401 12.245486 6.777278 4.776474 4.643057 5.317350
## [8] 7.095679 9.031722 11.000000
## [1] "Optimal model variable count:"
## [1] 6
## [1] "BICs: "
## [1] -1295.638 -1292.287 -1294.937 -1297.765 -1297.382 -1295.103 -1291.971
## [8] -1287.615 -1283.081 -1278.512
## [1] "Optimal model variable count:"
## [1] 4
## [1] "R-squared: "
## [1] 0.9999978 0.9999979 0.9999980 0.9999982 0.9999982 0.9999983 0.9999983
## [8] 0.9999983 0.9999983 0.9999983
## [1] "Optimal model variable count:"
## [1] 10Performing exhaustive subset selection via regsubsets, we see that the optimal model according to Cps is the 6 cofactor model with X to the power of 3, 5, 7, 8, 9, and 10. For tihs model, the Cp is at a minimum of 4.643057.
According to the BIC measure the optimal model is the 4 cofactor model with X to the power of 3, 5, 7, and 9. For this model, the BIC is at a minimum of -1297.765.
Similar to the previous problem, the optimal model according to the R-squared is unsuprisingly the model with the most amount of cofactors. Here, all models considered have a very large r-squared, since the one variable model sufficiently explains the response variable Y (since it was made with only one variable X^7).
f)
#fit lasso reg. model
x_dataM_2 = as.matrix(x_data_2)
lasso.mod_2=glmnet(x_dataM_2[,-11],x_dataM_2[ ,11],alpha=1, lambda = grid)
plot(lasso.mod_2, label=TRUE)
#Cross-validation
set.seed(42)
#x_dataf_2 = as.data.frame(x_data_2)
cv.out_2=cv.glmnet(x_dataM_2[,-11],x_dataM_2[ ,11],alpha=1)
plot(cv.out_2)
#find lambda corresponding to smallest mse
bestlam_2 =cv.out_2$lambda.min
bestlam_2## [1] 19.60987#extract coeficients for model with this lambda
coef(cv.out_2, s = "lambda.min")## 11 x 1 sparse Matrix of class "dgCMatrix"
## 1
## (Intercept) -1.958884
## X .
## X.2 .
## X.3 .
## X.4 .
## X.5 .
## X.6 .
## X.7 2.419947
## X.8 .
## X.9 .
## X.10 .The lambda that gives the smallest cross-validated mean-squared error is 19.60987. This lambda corresponds with the point on the cross-validation graph whose log(Lambda) = log(19.60987) = 1.2924, which is the point between the two calculated vertical dotted lines. The coefficient values can be found in the 11 x 1 spase matrix above. Notice that X^7 is the only covariate with non-zero coefficients, and it coefficient is 2.5. Remember our data was created using the real coefficient valeu of 2.5. Thus, this model is accurate in its assumptions, though it underestimates the X^2 coefficient. Again, the fact that this particular covariate has a non-zero coeficient in the optimal model is not suprising, since our data was generated using exclusively this X^7 variable multiplied by 2.5.
a)
indexes = sample(1:nrow(uni), size=0.5*nrow(uni))
test = uni[indexes,]
train = uni[-indexes,]b)
uniTrainingLM <- lm(Apps ~ ., train)
summary(uniTrainingLM)##
## Call:
## lm(formula = Apps ~ ., data = train)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3249.2 -497.2 -82.7 368.1 6922.8
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -2.712e+02 6.109e+02 -0.444 0.657341
## PrivateYes -7.848e+02 2.131e+02 -3.682 0.000265 ***
## Accept 1.263e+00 7.158e-02 17.646 < 2e-16 ***
## Enroll 7.895e-02 2.844e-01 0.278 0.781475
## Top10perc 5.412e+01 7.620e+00 7.102 6.32e-12 ***
## Top25perc -1.387e+01 6.431e+00 -2.158 0.031602 *
## F.Undergrad 2.522e-03 4.889e-02 0.052 0.958883
## P.Undergrad 4.884e-03 5.655e-02 0.086 0.931222
## Outstate -3.523e-02 2.886e-02 -1.221 0.222874
## Room.Board 2.259e-01 7.154e-02 3.157 0.001723 **
## Books -2.342e-02 3.523e-01 -0.066 0.947022
## Personal -7.133e-03 9.378e-02 -0.076 0.939410
## PhD -5.904e+00 7.265e+00 -0.813 0.416918
## Terminal -6.962e+00 8.137e+00 -0.856 0.392780
## S.F.Ratio 1.459e+00 1.840e+01 0.079 0.936845
## perc.alumni -1.103e+01 6.294e+00 -1.753 0.080418 .
## Expend 4.928e-02 1.626e-02 3.031 0.002611 **
## Grad.Rate 9.541e+00 4.388e+00 2.175 0.030295 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1098 on 371 degrees of freedom
## Multiple R-squared: 0.9216, Adjusted R-squared: 0.918
## F-statistic: 256.6 on 17 and 371 DF, p-value: < 2.2e-16#Calculate MSE on training set
mean(residuals(uniTrainingLM)^2)## [1] 1150171#Calculate MSE on test set by calculating residuals
mean(((test$Apps)-predict(uniTrainingLM,test))^2)## [1] 1209637The mean square error of the linear model is 1.209636610^{6}
c)
#Convert data to format usable by glmnet package, split into appropriate test and training data sets
x=model.matrix(Apps~.,uni)[,-1]
xTest = x[indexes,]
xTrain = x[-indexes,]
y=uni$Apps
yTest = uni[indexes,]$Apps
yTrain = uni[-indexes,]$Appsridge.cv = cv.glmnet(xTrain,yTrain,alpha=0)
lambda.cv = ridge.cv$lambda.min
lambda.cv## [1] 392.5473fit.ridge = glmnet(xTrain,yTrain,alpha=0,lambda=lambda.cv)
pred.ridge = predict(fit.ridge,newx=xTest)
mean((yTest-pred.ridge)^2)## [1] 2039168The lambda minimized by cross-validation in the ridge regression model is 392.5473434, and the resulting test error is 2.03916810^{6}
d)
lasso.cv = cv.glmnet(xTrain,yTrain,alpha=1)
lambda.cv = lasso.cv$lambda.min
lambda.cv## [1] 11.17989fit.lasso = glmnet(xTrain,yTrain,alpha=1,lambda=lambda.cv)
pred.lasso = predict(fit.lasso,newx=xTest)
mean((yTest-pred.lasso)^2)## [1] 1227317The lambda minimized by cross-validation in the LASSO model is 11.1798891, and the resulting test error is 1.227316710^{6}
e)
fit.pcr = pcr(Apps~.,data=train,scale=TRUE,validation="CV")
summary(fit.pcr)## Data: X dimension: 389 17
## Y dimension: 389 1
## Fit method: svdpc
## Number of components considered: 17
##
## VALIDATION: RMSEP
## Cross-validated using 10 random segments.
## (Intercept) 1 comps 2 comps 3 comps 4 comps 5 comps 6 comps
## CV 3841 3872 1900 1897 1628 1480 1451
## adjCV 3841 3878 1896 1895 1501 1474 1447
## 7 comps 8 comps 9 comps 10 comps 11 comps 12 comps 13 comps
## CV 1453 1449 1350 1339 1342 1342 1343
## adjCV 1450 1459 1336 1334 1337 1338 1339
## 14 comps 15 comps 16 comps 17 comps
## CV 1336 1342 1184 1185
## adjCV 1331 1338 1178 1179
##
## TRAINING: % variance explained
## 1 comps 2 comps 3 comps 4 comps 5 comps 6 comps 7 comps
## X 31.371 57.65 65.46 71.25 76.87 81.56 85.11
## Apps 1.626 76.68 76.87 86.56 86.72 87.01 87.01
## 8 comps 9 comps 10 comps 11 comps 12 comps 13 comps 14 comps
## X 88.03 90.81 93.33 95.41 97.19 98.17 98.90
## Apps 87.21 89.06 89.20 89.20 89.23 89.24 89.37
## 15 comps 16 comps 17 comps
## X 99.48 99.86 100.00
## Apps 89.40 92.10 92.16pcr_pred <- predict(fit.pcr, test)
mean((pcr_pred - test$Apps)^2)## [1] 3623155validationplot(fit.pcr,val.type="MSEP")
Appears the smallest Cross Validation error occurs when all components are included in the model (M = 17)
f)
mean(((test$Apps)-predict(uniTrainingLM,test))^2) # Least Squares MSE## [1] 1209637mean((yTest-pred.ridge)^2) # Ridge MSE## [1] 2039168mean((yTest-pred.lasso)^2) # LASSO MSE## [1] 1227317mean((pcr_pred - test$Apps)^2) # PCR MSE## [1] 3623155coef(uniTrainingLM)## (Intercept) PrivateYes Accept Enroll Top10perc
## -2.712196e+02 -7.848468e+02 1.263133e+00 7.895531e-02 5.411756e+01
## Top25perc F.Undergrad P.Undergrad Outstate Room.Board
## -1.387474e+01 2.522176e-03 4.884264e-03 -3.523260e-02 2.258838e-01
## Books Personal PhD Terminal S.F.Ratio
## -2.342423e-02 -7.132978e-03 -5.903813e+00 -6.961877e+00 1.458547e+00
## perc.alumni Expend Grad.Rate
## -1.103375e+01 4.928336e-02 9.540876e+00coef(fit.ridge)## 18 x 1 sparse Matrix of class "dgCMatrix"
## s0
## (Intercept) -1.310206e+03
## PrivateYes -6.941615e+02
## Accept 7.606190e-01
## Enroll 7.650760e-01
## Top10perc 2.987822e+01
## Top25perc 2.016432e+00
## F.Undergrad 9.180116e-02
## P.Undergrad -1.187016e-03
## Outstate 1.111003e-02
## Room.Board 2.669129e-01
## Books 2.647735e-02
## Personal -6.344657e-02
## PhD -9.017944e-01
## Terminal -8.043214e+00
## S.F.Ratio 4.032825e+00
## perc.alumni -1.640348e+01
## Expend 5.410772e-02
## Grad.Rate 1.052146e+01coef(fit.lasso)## 18 x 1 sparse Matrix of class "dgCMatrix"
## s0
## (Intercept) -3.849310e+02
## PrivateYes -7.627745e+02
## Accept 1.251083e+00
## Enroll 9.984464e-02
## Top10perc 4.803665e+01
## Top25perc -8.839646e+00
## F.Undergrad 2.877663e-03
## P.Undergrad .
## Outstate -2.213359e-02
## Room.Board 2.001993e-01
## Books .
## Personal .
## PhD -4.473273e+00
## Terminal -6.889357e+00
## S.F.Ratio .
## perc.alumni -1.076730e+01
## Expend 4.575306e-02
## Grad.Rate 7.870912e+00coef(fit.pcr)## , , 17 comps
##
## Apps
## PrivateYes -351.932393
## Accept 3164.692435
## Enroll 78.990563
## Top10perc 1001.943589
## Top25perc -282.169693
## F.Undergrad 13.136146
## P.Undergrad 6.921946
## Outstate -145.545916
## Room.Board 259.997785
## Books -3.995510
## Personal -4.882939
## PhD -98.764668
## Terminal -102.093346
## S.F.Ratio 5.981821
## perc.alumni -138.326075
## Expend 277.576700
## Grad.Rate 164.551336We can measure the accuracy of a model through the Mean Squared Error calculated using a separate test set. This measures the average squared residual and can assess how well a model fits a data set, in these cases test sets. The Least Squares Regression and the LASSO model both tend to have similar test errors, which are the lowest. The ridge regression model tends to have higher test error, with the PCR model having the worst test error on average. If these procedures were repeated for a different split between training and test sets, the trends usually hold but the values themselves change.
attach(Boston)
head(Boston)## crim zn indus chas nox rm age dis rad tax ptratio black
## 1 0.00632 18 2.31 0 0.538 6.575 65.2 4.0900 1 296 15.3 396.90
## 2 0.02731 0 7.07 0 0.469 6.421 78.9 4.9671 2 242 17.8 396.90
## 3 0.02729 0 7.07 0 0.469 7.185 61.1 4.9671 2 242 17.8 392.83
## 4 0.03237 0 2.18 0 0.458 6.998 45.8 6.0622 3 222 18.7 394.63
## 5 0.06905 0 2.18 0 0.458 7.147 54.2 6.0622 3 222 18.7 396.90
## 6 0.02985 0 2.18 0 0.458 6.430 58.7 6.0622 3 222 18.7 394.12
## lstat medv
## 1 4.98 24.0
## 2 9.14 21.6
## 3 4.03 34.7
## 4 2.94 33.4
## 5 5.33 36.2
## 6 5.21 28.7#Initial Summary
summary(Boston)## crim zn indus chas
## Min. : 0.00632 Min. : 0.00 Min. : 0.46 Min. :0.00000
## 1st Qu.: 0.08204 1st Qu.: 0.00 1st Qu.: 5.19 1st Qu.:0.00000
## Median : 0.25651 Median : 0.00 Median : 9.69 Median :0.00000
## Mean : 3.61352 Mean : 11.36 Mean :11.14 Mean :0.06917
## 3rd Qu.: 3.67708 3rd Qu.: 12.50 3rd Qu.:18.10 3rd Qu.:0.00000
## Max. :88.97620 Max. :100.00 Max. :27.74 Max. :1.00000
## nox rm age dis
## Min. :0.3850 Min. :3.561 Min. : 2.90 Min. : 1.130
## 1st Qu.:0.4490 1st Qu.:5.886 1st Qu.: 45.02 1st Qu.: 2.100
## Median :0.5380 Median :6.208 Median : 77.50 Median : 3.207
## Mean :0.5547 Mean :6.285 Mean : 68.57 Mean : 3.795
## 3rd Qu.:0.6240 3rd Qu.:6.623 3rd Qu.: 94.08 3rd Qu.: 5.188
## Max. :0.8710 Max. :8.780 Max. :100.00 Max. :12.127
## rad tax ptratio black
## Min. : 1.000 Min. :187.0 Min. :12.60 Min. : 0.32
## 1st Qu.: 4.000 1st Qu.:279.0 1st Qu.:17.40 1st Qu.:375.38
## Median : 5.000 Median :330.0 Median :19.05 Median :391.44
## Mean : 9.549 Mean :408.2 Mean :18.46 Mean :356.67
## 3rd Qu.:24.000 3rd Qu.:666.0 3rd Qu.:20.20 3rd Qu.:396.23
## Max. :24.000 Max. :711.0 Max. :22.00 Max. :396.90
## lstat medv
## Min. : 1.73 Min. : 5.00
## 1st Qu.: 6.95 1st Qu.:17.02
## Median :11.36 Median :21.20
## Mean :12.65 Mean :22.53
## 3rd Qu.:16.95 3rd Qu.:25.00
## Max. :37.97 Max. :50.00#remove factor like variables
Boston_cont = subset(Boston, select=-c(rad, chas))
#Pairplot
pairs(Boston_cont, cex = .5)
#Boxplot for factor var. rad
boxplot(crim~rad, main="Crime Rate by Rad",
xlab="Rad", ylab="Crime Rate")
#Boxplot for factor var. chas
boxplot(crim~chas, main="Crime Rate by Chas",
xlab="Chas", ylab="Crime Rate")
Boston_to_trans = subset(Boston, select=c(crim,age, dis, medv))
#before transform
pairs(Boston_to_trans)
#transform
Boston_to_trans[,2:4] = log(Boston_to_trans[,2:4])
#after transform
pairs(Boston_to_trans)
#zn indus tax and ptratio all seem to only have high crime rates at one value. See which value that is
boxplot(crim~zn, main="Crime Rate by zn", xlab="zn", ylab="Crime Rate")
boxplot(crim~indus, main="Crime Rate by indus", xlab="indus", ylab="Crime Rate")
boxplot(crim~tax, main="Crime Rate by tax", xlab="tax", ylab="Crime Rate")
boxplot(crim~ptratio, main="Crime Rate by ptratio", xlab="ptratio", ylab="Crime Rate")
#zn = 0 , indus = 18.10 , Looking at the pairs plot, zn, indus, age, dis, tax, and ptratio seem to be the strongest possible contenders for covariates to be included in our model. This is of course based on their correlation with the crim variable (the top row). For the two factor covariates rad and chas, rad value of 24 seems to be correlated with larger crime rate and while the mean crim rate of chas=0 points is low, any point with a large crime rate had a chas=0. In orther words, very little of the chas=1 points had a large crime rate.
#fit lasso reg. model
Boston_M = as.matrix(Boston)
lambdas=seq(1e-3,1e3,length=100)
lasso.mod_3=glmnet(Boston_M[,-1],Boston_M[ ,1],alpha=1, lambda = lambdas)
plot(lasso.mod_3, label=TRUE)
#Cross-validation
set.seed(42)
#Boston_df = as.data.frame(Boston_df)
cv.out_3=cv.glmnet(Boston_M[,-1],Boston_M[ ,1],alpha=1)
plot(cv.out_3)
#find lambda corresponding to smallest mse
bestlam =cv.out_3$lambda.min
bestlam## [1] 0.05630926#extract coeficients for model with this lambda
coef(cv.out_3, s = "lambda.min")## 14 x 1 sparse Matrix of class "dgCMatrix"
## 1
## (Intercept) 12.319178096
## zn 0.035726832
## indus -0.068876055
## chas -0.577832639
## nox -6.631559478
## rm 0.208676938
## age .
## dis -0.768388825
## rad 0.512333871
## tax .
## ptratio -0.179631375
## black -0.007551172
## lstat 0.124630014
## medv -0.154550130x = Boston_M[,-1]
y=Boston_M[ ,1]
finalModel1=glmnet(x,y,alpha=1, lambda = 0.05630926)
#plotres(finalModel1, which=1)
pred.lasso = predict(finalModel1,newx=x)
mean((y-pred.lasso)^2)## [1] 40.48466summary(finalModel1)## Length Class Mode
## a0 1 -none- numeric
## beta 13 dgCMatrix S4
## df 1 -none- numeric
## dim 2 -none- numeric
## lambda 1 -none- numeric
## dev.ratio 1 -none- numeric
## nulldev 1 -none- numeric
## npasses 1 -none- numeric
## jerr 1 -none- numeric
## offset 1 -none- logical
## call 5 -none- call
## nobs 1 -none- numericfits <- fitted(finalModel1)
## calculate the deviance residuals
resids <- (pred.lasso-y)
fits## NULLplot(pred.lasso, resids, main = "Model Resids")
abline(h=0, lty=2)
#QQ to see if residuals follow normal
qqnorm(y-pred.lasso)
qqline(y-pred.lasso)
We use cross validation via the ‘cv.glmnet’ funciton in order to validate our model selection and paremeter (lambda) setting. The minimizing lambda for this lasso regression is 0.0563092. This lambda corresponds with the point on the cross-validation graph whose log(Lambda) = log(0.0563092.) = -1.24942, which is the point between the two calculated vertical dotted lines. The covariates determined not useful and thus whose coefficients are set to zero in the model are age and tax. While these two variables looked as though they may have been correlated with crime rate, they could have been left out of the model due to multicollinearity. We validated our model using cross validation and considered a large amount of lambdas as values. The optimal lambda fell comfortably in between the max and min of all the lambas considered.
The residuals are not optmial, but they do have decent amounts both above and below the y=0 line. They also, for the majority of the middle instances, follow the qq line in the last plot. They only stray from it in the early and the late theoretical quantiles.